ENMA: Tokenwise Autoregression for Generative Neural PDE Operators
Armand Kassaï Koupaï, Lise Le Boudec, Louis Serrano, Patrick Gallinari · 2025
Read ENMA: Tokenwise Autoregression for Generative Neural PDE Operators by Armand Kassaï Koupaï, Lise Le Boudec, Louis Serrano, Patrick Gallinari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06158.txt.
Introduction
Armand Kassai Koupai and co-authors build a generative solver for physics equations that evolve in time in a 2025 paper. Neural solvers struggle to handle wide parameter ranges and patchy data. Their model, ENMA, works in a squeezed hidden space where future states are grown token by token.
Spotty spatial samples are first coded into even hidden grids by attention, then squeezed further by a layered encoder. At run time the model can lean on past states or on like past runs as context. One-shot fits to fresh equation settings follow from that trick. The frame suits forecasting of flows, waves, and other spread-out dynamics.
Catalog record
| CREATOR | Armand Kassaï Koupaï, Lise Le Boudec, Louis Serrano, Patrick Gallinari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 116 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ENMA: Tokenwise Autoregression for Generative Neural PDE Operators Armand Kassaï Koupaï, Lise Le Boudec, Louis Serrano, Patrick Gallinari Abstract Solving time-dependent parametric partial differential equations (PDEs) remains a fundamental challenge for neural solvers, particularly when generalizing across a wide range of physical parameters and dynamics. When data is uncertain or incomplete—as is often the case—a natural approach is to turn to generative models. We introduce ENMA , a generative neural operator designed to model spatio-temporal dynamics arising from physical phenomena. ENMA predicts future dynamics in a compressed latent space using a generative masked autoregressive transformer trained with flow matching loss, enabling tokenwise generation . Irregularly sampled spatial observations are encoded into uniform latent representations via attention mechanisms and further compressed through a spatio-temporal convolutional encoder. This allows ENMA to perform in-context learning at inference time by conditioning on either past states of the target trajectory or auxiliary context trajectories with similar dynamics. The result is a robust and adaptable framework that generalizes to new PDE regimes and supports one-shot surrogate modeling of time-dependent parametric PDEs. 1 Introduction Neural surrogates for spatio-temporal dynamics and PDE solving have emerged as efficient alternatives to traditional numerical solvers, driving rapid advances in the field. Early work ( de Bezenac et al. 2018 ; Long et al. 2018 ; Raissi et al. 2019 ; Wiewel et al. 2019 ) laid the groundwork, followed by a wave of models leveraging diverse architectural designs ( Yin et al. 2022b ; Brandstetter et al. 2022 ; Wu et al. 2024a ; Ayed et al. 2022 ) . Neural operators (NOs) ( Chen & Chen 1995 ; Li et al. 2020a ; Lu et al. 2021 ) expanded the paradigm by learning mappings between infinite-dimensional function spaces. Recent models have widely adopted this framework, improving scalability and flexibility ( Gupta et al. 2021 ; Hao et al. 2023 ; Alkin et al. 2024a ; Serrano et al. 2024c ; Wu et al. 2024a ) . While earlier approaches focused on fixed PDE instances, current research tackles the more general task of learning parametric PDEs ( Kirchmeyer et al. 2022 ; Nzoyem et al. 2024 ; Koupaï et al. 2024 ) , and is now moving toward foundation models capable of handling multi-physics regimes ( Subramanian et al. 2023 ; McCabe et al. 2023 ; Liu et al. 2025 ; Cao et al. 2024 ; Morel & Oyallon 2025 ) . Most neural PDE solvers to date have focused on learning deterministic mappings, limiting their ability to capture complex or uncertain physical behaviors. This has spurred interest in stochastic modeling through generative probabilistic methods. A primary motivation arises from chaotic systems like weather forecasting ( Price et al. 2025 ; Couairon et al. 2024 ) and turbulent flows ( Kohl et al. 2024 ) . Another challenge is error accumulation in autoregressive models which hampers long-term predictions and could be mitigated through probabilistic forecasters ( Lippe et al. 2023 ; Kohl et al. 2024 ) . More generally, data-driven surrogates must handle both aleatoric and epistemic uncertainty ( Bülte et al. 2025 ; Wu et al. 2024b ) , often under conditions of partial observability, sensor noise, or coarse space-time resolution. Additionally, they face distribution shifts at test time, where PDE parameters may deviate from training data. Epistemic uncertainty may also stem from model mismatch or limited training data, reducing generalization to out-of-distribution regimes ( Mouli et al. 2024 ) . Generative neural surrogates for time-dependent PDEs increasingly draw on advances from computer vision and, more recently, language modeling. These approaches typically fall into two categories. The first includes diffusion models , which operate in continuous spaces—pixel-level or latent—and model the joint distribution of future states by reversing a noising process. Based on U-Nets or transformer encoders ( Ho et al. 2020 ; Peebles & Xie 2022 ) , they have been extended to PDE forecasting via iterative denoising ( Zhou et al. 2025 ; Lippe et al. 2023 ; Kohl et al. 2024 ; Li et al. 2025 ) . The second family follows an autoregressive (AR) paradigm, predicting per-token conditional distributions ( Touvron et al. 2023 ) . These models discretize physical fields using vector quantization ( Oord et al. 2017 ; Mentzer et al. 2024 ) , and generate sequences either sequentially ( Yu et al. 2024 ) or via masked decoding ( He et al. 2022 ; Yu et al. 2023 ) , with recent adaptations for PDE surrogates and in-context learning ( Serrano et al. 2024a ) . Each approach comes with trade-offs. Diffusion models are robust to distribution drift and support uncertainty quantification via noise injection, but require expensive training and inference ( Leng et al. 2025 ) . AR models offer efficient, causality-aligned generation via KV caching and support in-context learning ( Brown et al. 2020 ) , but suffer from reduced expressiveness due to discretization and often yield uncalibrated uncertainty estimates ( Jiang et al. 2020 ) . They are also more sensitive to error accumulation from teacher forcing. Recently, continuous-token AR models have emerged as a promising alternative to discrete decoding in vision ( Li et al. 2025 ) , motivating their application to scientific modeling. Figure 1 : ENMA: Continuous Spatio-Temporal Autoregressive Generation. Given an initial sequence of latent states \bm{Z}^{0:L-1} , a causal transformer predicts the next latent frame \bm{Z_{\texttt{DYN}}}^{L} using block-wise attention for temporal autoregression (Sec. 3.1.1 ). This state is concatenated with masked tokens and passed to a spatial transformer, which performs masked spatial autoregression by progressively decoding tokens across multiple steps. A lightweight MLP trained with flow matching produces per-token predictions conditioned on spatial transformer predictions (Sec. 2 ). The completed frame \hat{\bm{Z}}^{L} is appended to the history for rollout. The full latent sequence is then decoded into the physical domain using a continuous VAE decoder (Sec. 3.2 ). We address fundamental limitations in existing generative surrogate models for PDEs—particularly their reliance on discrete tokenization or full-frame diffusion, which hampers scalability, uncertainty modeling, and physical consistency. We introduce ENMA (presented in Figure 1 ), a continuous autoregressive neural operator for modeling time-dependent parametric PDEs , where parameters such as initial conditions, coefficients, and forcing terms may vary across instances. ENMA operates entirely in a continuous latent space and advances both the encoder-decoder pipeline and the generative modeling component crucial to neural PDE solvers. The encoder employs attention mechanisms to process irregular spatio-temporal inputs (i.e., unordered point sets) and maps them onto a structured, grid-aligned latent space. A causal spatio-temporal convolutional encoder ( Yu et al. 2024 ) then compresses these observations into compact latent tokens spanning multiple states. Generation proceeds in two stages. A causal transformer first predicts future latent states autoregressively. Then, a masked spatial transformer decodes each state at the token level using Flow Matching ( Lipman et al. 2023 ; Lipman et al. 2024 ) to model per-token conditional distributions in continuous space—providing a more efficient alternative to full-frame diffusion ( Lippe et al. 2023 ; Kohl et al. 2024 ; Li et al. 2025 ) . Finally, the decoder reconstructs the full physical trajectory from the generated latents. ENMA improves state-of-the-art generative surrogates by combining flexible per-token generation with expressive and compact