VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models
Edward Li, Zichen Wang, Jiahe HuangJeong Joon ParkUniversity of Michigan, Ann Arbor {edwarli, zzzichen, chloehjh, jjparkcv}@umich.edu · 2025
Read VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models by Edward Li, Zichen Wang, Jiahe HuangJeong Joon ParkUniversity of Michigan, Ann Arbor {edwarli, zzzichen, chloehjh, jjparkcv}@umich.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13754.txt.
Introduction
Edward Li, Zichen Wang, Jiahe Huang, and Jeong Joon Park solve physics equations by treating them as videos with missing frames. Partial differential equations describe heat, fluids, and waves, and solvers usually split into forward prediction and inverse guessing. Their 2025 framework, VideoPDE, handles both in one generative model.
The trick is inpainting: given some known patches of space and time, the model fills the rest. A future forecast becomes filling in frames after the known start; an inverse problem fills gaps elsewhere. A diffusion transformer works in raw pixel space for fine detail, with a layered design for speed. Across many equations and setups, the method beat leading baselines for accuracy and flexibility.
Catalog record
| CREATOR | Edward Li, Zichen Wang, Jiahe HuangJeong Joon ParkUniversity of Michigan, Ann Arbor {edwarli, zzzichen, chloehjh, jjparkcv}@umich.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13754
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models Edward Li, Zichen Wang, Jiahe HuangJeong Joon ParkUniversity of Michigan, Ann Arbor {edwarli, zzzichen, chloehjh, jjparkcv}@umich.edu Abstract We present a unified framework for solving partial differential equations (PDEs) using video-inpainting diffusion transformer models. Unlike existing methods that devise specialized strategies for either forward or inverse problems under full or partial observation, our approach unifies these tasks under a single, flexible generative framework. Specifically, we recast PDE-solving as a generalized inpainting problem, e.g., treating forward prediction as inferring missing spatiotemporal information of future states from initial conditions. To this end, we design a transformer-based architecture that conditions on arbitrary patterns of known data to infer missing values across time and space. Our method proposes pixel-space video diffusion models for fine-grained, high-fidelity inpainting and conditioning, while enhancing computational efficiency through hierarchical modeling. Extensive experiments show that our video inpainting-based diffusion model offers an accurate and versatile solution across a wide range of PDEs and problem setups, outperforming state-of-the-art baselines. Project page: videopde.github.io . 1 Introduction Computational science communities have proposed numerous learning-based approaches for solving PDE-governed systems for simulation, optimization, or scientific discovery. These methods provide tradeoffs across accuracy, applicability, and speed. Physics-informed neural networks (PINNs) raissi2019physics ; raissi2017physics can be flexibly applied to forward/inverse predictions or sparse differential measurements, but their optimization often falls into a local minimum, sacrificing accuracy. Neural Operators li2020fourier ; li2021physics ; lu2021learning offer fast approximate simulations, but struggle to handle partial observations common in real-world problems. Recent generative methods chen2023seine ; Shu_2023 ; zhuang2024spatiallyawarediffusionmodelscrossattention accommodate partial observations, at the expense of slow speed and inability to model dense temporal states. These challenges have limited the real-world applicability of learning-based PDE approaches for past state reconstructions or future simulations. In this work, we introduce VideoPDE, a unified framework that is accurate, fast, and applicable to a diverse range of scenarios. Intuitively, VideoPDE casts the problem of PDE solving in diverse settings as a generative video inpainting problem, leveraging the power of diffusion models. For example, forward simulation can be viewed as predicting the missing pixels in frames 2 to T, given full or partial observations of the initial frame. This generative framework unifies diverse sensor configurations, supports non-deterministic predictions for chaotic systems, and offers a fast yet accurate alternative to classical solvers—all within a single neural network. Our method uses a transformer-based video diffusion model (VDM) ho2022videodiffusionmodels architecture that can be conditioned on arbitrary spatiotemporal patterns of partial measurements. Unlike common VDMs that operate in the latent space blattmann2023videoldm ; gupta2023photorealisticvideogenerationdiffusion ; he2022lvdm ; zhou2023magicvideoefficientvideogeneration , our method can denoise and condition at the pixel level, well-suited for scientific applications that require fine-grained accuracy, rather than perceptual realism. While VDMs have been used for video inpainting and prediction in the natural video domain hoppe2022diffusion ; lu2023vdtgeneralpurposevideodiffusion ; voleti2022MCVD ; zhang2023avid , no prior works (scientific or non-scientific domain) have tried hierarchical pixel-level modeling and intra-token conditioning that lead to exceptional accuracy and efficiency. Our contributions include: (i) the formulation of casting PDE solving as generative video inpainting, (ii) architectural innovation that leads to efficient pixel-space denoising and conditioning, (iii) empirical demonstration of generative PDE solving across diverse settings. Our trained models obtain state-of-the-art results across settings, producing accurate predictions from as little as 1% continuous measurements and reducing error by up to an order of magnitude compared to prior methods. 2 Related Work Neural PDEs Solving partial differential equations (PDEs) is a fundamental problem in physical sciences. Traditional numerical methods such as the Finite Element Method quarteroni2008numerical ; solin2005partial and Boundary Element Method aliabadi2020boundary ; idelsohn2003meshless have long been the backbone of PDE solving but are computationally expensive and inflexible for complex systems. Neural approaches offer data-driven alternatives: physics-informed neural networks (PINNs) raissi2017physics ; raissi2019physics enforce PDE constraints via the loss function and have been applied to a wide range of PDEs cai2021physicsfluid ; cai2021physicsheat ; eivazi2022physics ; hao2023pinnacle ; mao2020physics ; misyris2020physics ; PENWARDEN2023111912 ; toscano2024pinnspikansrecentadvances ; Waheed_2021 . While PINNs can work on sparse measurements, in practice they often face optimization instability and poor scalability. Neural operators, such as FNO li2020fourier , DeepONet lu2021learning , and PINO li2021physics , learn mappings between function spaces to avoid expensive optimization and achieve resolution-invariance. These models have been extended to various forward bonev2023spherical ; chetan2023accurate ; li2023solving ; li2022fourier ; li2024geometry ; peng2023linear ; serrano2024operator ; wen2022u and inverse long2024invertible ; molinaro2023neural PDE tasks, but remain limited in flexibility for handling arbitrary and sparse input patterns. Solving PDEs Under Sparse Measurements Recently, neural methods have gained attention for solving PDEs under sparse measurements, reflecting the challenge of acquiring full spatiotemporal data. DiffusionPDE huang2024diffusionpdegenerativepdesolvingpartial addresses this by modeling the joint distribution over coefficients and solutions, allowing flexible inputs, but its DPS chung2024diffusionposteriorsamplinggeneral backbone requires PDE-specific tuning and struggles with dynamic PDEs. Spatially-aware diffusion models zhuang2024spatiallyawarediffusionmodelscrossattention use cross-attention to handle partial observations but lack temporal modeling. Temporal PDEs are especially important for modeling nonlinear fluid and gas dynamics darman2025fourieranalysisphysicstransfer ; dheeshjith2024samudraaiglobalocean ; kashinath2021physics ; wang2020towards ; zong2023neural . Super-resolution frameworks Fukami_2020 ; fukami2023super ; Kim_2021 reconstruct full fields from coarse data. Recent methods li2023transformer ; shan2024pirdphysicsinformedresidualdiffusion ; Shu_2023 combine physics-informed losses with diffusion models or transformers for high-fidelity turbulent flow reconstruction. Despite past successes, existing methods often rely on strong assumptions about PDEs, boundary conditions, or sensor layouts. We propose a unified generative framework that requires no prior knowledge and generalizes well across forward, inverse, and partial observation problems. Inpainting Diffusion Models Diffusion models ho2020denoising ; sohldickstein2015deepunsupervisedlearningusing ; song2020generativemodelingestimatinggradients ; song2021scorebasedgenerativemodelingstochastic ; song2022diffusionsurvey have emerged as particularly suited for image and video inpainting due to their capability to model complex, high-dimensional distributions effectively. Training-free methods guide the sampling trajectory to satisfy the conditions at inference time through noise inference meng2022sdedit , resampling lugmayr2022repaintinpaintingusingdenoising , or latent alignment choi2021ilvrconditioningmethoddenoising . It can also be studied as a linear inverse problem chung2024diffusionposteriorsamplinggeneral ; chung2024improvingdiffusionmodelsinverse ; kawar2022denoising ; song2024solvinginverseproblemslatent ; xu2023infedit . However, these methods often struggle with extremely sparse or ambiguous observations. Another class of methods directly trains a conditional diffusion model. These methods typically modify the network architecture to inject conditioning information, such as channel concatenation saharia2022paletteimagetoimagediffusionmodels ; saharia2021imagesuperresolutioniterativerefinement , cross-attention balaji2023ediffitexttoimagediffusionmodels ; perez2017filmvisualreasoninggeneral ; rombach2022high , or ControlNet zhang2023addingconditionalcontroltexttoimage . We adopt channel concatenation in this work since it is simple and effective. These conditioning techniques have been extended to video diffusion models ho2022videodiffusionmodels for video inpainting lu2024vdt ; wang2023videocomposercompositionalvideosynthesis ; zhang2024avidanylengthvideoinpainting . Figure 1 : VideoPDE pipeline. We cast PDE solving as a video inpainting task. Our Hierarchical Video Diffusion Transformer (HV-DiT) denoises initial noise into a full video, conditioned on pixel-level sparse measurements. Its ability to handle arbitrary input patterns enables flexible application to diverse PDE scenarios, including forward, inverse, and continuous measurement tasks. 3 Methods 3.1 Preliminaries: Diffusion Models and Guided Sampling Diffusion models learn data distributions by reversing a gradual noising process. Starting from a clean sample \mathbf{x}_{0} from a data distribution p(\mathbf{x}) , a forward stochastic process progressively adds noise \epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}) to produce \mathbf{x}_{t}=\mathbf{x}_{0}+\sigma(t)\epsilon and hence a family of distributions p(\mathbf{x}_{t};\sigma(t)) , where \sigma(t) denotes the standard deviation of the noise at diffusion time t , following the noise schedule of the Elucidating Diffusion Models