Beyond Loss Guidance: Using PDE Residuals as Spectral Attention in Diffusion Neural Operators
Medha Sawhney, Abhilash Neog, Mridul Khurana & Anuj Karpatne · 2025
Read Beyond Loss Guidance: Using PDE Residuals as Spectral Attention in Diffusion Neural Operators by Medha Sawhney, Abhilash Neog, Mridul Khurana & Anuj Karpatne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01370.txt.
Introduction
Medha Sawhney and co-authors speed up diffusion solvers for physics equations. These solvers guide each step with the leftover error of the equation. That guide needs slow gradient work at test time. It can also wobble when the error signal is noisy.
The team uses the error in a new way. They treat it as a form of attention in the frequency domain. That lets the solver adapt its steps on the fly. It skips the slow guide loop as well. Faster, steadier solvers help engineers run many what-if tests on flows, heat, and stress.
Catalog record
| CREATOR | Medha Sawhney, Abhilash Neog, Mridul Khurana & Anuj Karpatne |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01370
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Loss Guidance: Using PDE Residuals as Spectral Attention in Diffusion Neural Operators Medha Sawhney, Abhilash Neog, Mridul Khurana & Anuj Karpatne Abstract Diffusion-based solvers for partial differential equations (PDEs) are often bottle-necked by slow gradient-based test-time optimization routines that use PDE residuals for loss guidance. They additionally suffer from optimization instabilities and are unable to dynamically adapt their inference scheme in the presence of noisy PDE residuals. To address these limitations, we introduce PRISMA (PDE Residual Informed Spectral Modulation with Attention), a conditional diffusion neural operator that embeds PDE residuals directly into the model’s architecture via attention mechanisms in the spectral domain, enabling gradient-descent free inference. In contrast to previous methods that use PDE loss solely as external optimization targets, PRISMA integrates PDE residuals as integral architectural features, making it inherently fast, robust, accurate, and free from sensitive hyperparameter tuning. We show that PRISMA has competitive accuracy, at substantially lower inference costs , compared to previous methods across five benchmark PDEs especially with noisy observations, while using 10x to 100x fewer denoising steps, leading to 15x to 250x faster inference. 1 Introduction Given the ubiquitous presence of partial differential equations (PDEs) in almost every scientific discipline, there is a rapidly growing literature on using neural networks for solving PDEs ( Raissi et al. 2019a ; Lu et al. 2019 ) . This includes seminal works in operator learning methods such as the Fourier Neural Operator (FNO) Li et al. 2020 that learns resolution-independent mappings between function spaces of input parameters \mathbf{a} and solution fields \mathbf{u} . However, a major limitation of these methods is their reliance on complete and clean observations of either \mathbf{a} or \mathbf{u} , a condition rarely met in real-world applications where data is inherently noisy and sparse. The rise of generative models has inspired another class of methods for solving PDEs by modeling the joint distribution of \mathbf{a} and \mathbf{u} using diffusion-based backbones ( Huang et al. 2024 ; Yao et al. 2025 ; Lim et al. 2023 ; Shu et al. 2023 ; Bastek et al. 2024 ; Jacobsen et al. 2025 ) . These methods offer two key advantages over operator learning methods: (i) they generate full posterior distributions of \mathbf{a} and/or \mathbf{u} , enabling principled uncertainty quantification crucial for ill-posed inverse problems, and (ii) they naturally accommodate sparse observations during inference using likelihood-based and PDE residual-based loss guidance , termed diffusion posterior sampling or test-time optimization. Despite these advances, current diffusion-based PDE solvers suffer from three major limitations. First , they are inherently slow during inference due to the large number of denoising steps required by their unconditional training paradigm, which is further burdened by expensive test-time optimization when PDE-residual loss is optimized. Second, the minimization of PDE residuals as external loss introduces optimization instabilities due to the sensitive nature of PDE loss gradients at different stages of inference, requiring careful hyperparameter tuning ( Zhang & Zou 2025 ; Cheng et al. 2024 ; Utkarsh et al. 2025 ) . Third , when observations are corrupted by unknown noise structures (e.g., sensor failures or environmental interference), these methods cannot dynamically adapt their guided inference scheme to discount noisy PDE residuals, resulting in degraded performance. Figure 1: The PRISMA inference pipeline. We use a U-shaped Diffusion Neural Operator (UNO) to iteratively refine estimates of \mathbf{a} and \mathbf{u} starting from noise (\mathbf{a},\mathbf{u})_{T} to a clean solution (\mathbf{a},\mathbf{u})_{0} . At each denoising step, PDE residuals are architecturally injected via a novel SRA block at every layer of the UNO, enabling fast, gradient-descent free inference for both forward and inverse problems. At the heart of these limitations lies a fundamental bottleneck in existing generative frameworks for solving PDEs: “treating PDE constraints as external optimization targets rather than as integral architectural features .” Since previous methods employ PDE residuals solely as additional loss terms during training or inference, they face a challenging multi-objective optimization problem that is hard to stabilize and requires careful hyper-parameter tuning ( Wang et al. 2021 ; Farea et al. 2025 ) . Moreover, by reducing spatially-resolved PDE residual fields to scalar loss terms, existing methods discard the rich point-wise information that could otherwise provide local guidance for fine-grained steering. In contrast to using PDE residuals as loss guidance, we ask: “is it possible to incorporate the knowledge of PDE residuals directly in the architecture of diffusion-based models, avoiding slow and sensitive loss gradients during inference altogether?” We introduce PRISMA (PDE Residual Informed Spectral Modulation with Attention), a conditional diffusion neural operator that embeds PDE residuals into the architecture via attention mechanisms instead of loss guidance, enabling gradient-descent free inference . A key innovation in PRISMA is a novel Spectral Residual Attention ( SRA ) block (Figure 1 ) that is injected at every layer of a U-shaped denoiser backbone to locally attend to the model’s generated outputs based on guidance from the PDE residual field, during both training and inference. By integrating PDE residuals as architectural features directly in the spectral domain , SRA is able to modulate and dynamically adapt to the local contributions of varying frequency modes in the PDE residual space, even in the presence of noise. Another innovation in PRISMA is the use of input masks in the architecture to regulate the generation of \mathbf{a} and \mathbf{u} under varying conditions of input observations, rather than training an unconditional model guided only at inference. This conditional design enables PRISMA to serve as a unified framework for solving PDEs: a single versatile model capable of addressing both forward and inverse problems under full, sparse, or noisy observations, without task-specific inference. Our work makes four key contributions. (1) We introduce a novel strategy for incorporating physical knowledge in diffusion models as architectural guidance , where the PDE residual is embedded directly as an internal feature in the model instead of a test-time guidance loss. (2) A powerful outcome of this design is gradient-descent free inference , which fundamentally removes the need for costly and sensitive optimization of likelihood and PDE loss terms during inference, forming the basis of our method’s speed and robustness. (3) We show 15x to 250x faster inference (time) compared to baselines across a diverse set of benchmark PDEs with varying problem configurations (forward and inverse), while achieving superior accuracy especially in the presence of noisy observations. (4) Our work also represents the first conditional design of a generative model trained to unify forward and inverse problems under varying configurations of input observations. 2 Related Works Neural PDE solvers: Deep learning methods such as Physics-Informed Neural Networks (PINNs) ( Raissi et al. 2019b ) offer a powerful alternative to classical methods for solving PDEs by minimizing PDE residual loss during training. However, PINNs are prone to optimization and scaling challenges ( Zhongkai et al. 2024 ; Liu et al. 2024a ) , motivating operator-learning methods such as FNO and DeepONet, which learn resolution-invariant mappings between functions of input parameters and solution fields for fast inference ( Li et al. 2020 ; Rahman et al. 2022 ; Kovachki et al. 2023 ; Lu et al. 2019 ) . To further enhance physics guidance in FNO architecture, PINO ( Li et al. 2024 ) incorporates physics-based loss during training and