Representation Learning for Spatiotemporal Physical Systems

Helen Qu 1 Rudy Morel 1 Michael McCabe 1,4 Alberto Bietti 1 François Lanusse 2 Shirley Ho 1,3,4 Yann LeCun 3 The Polymathic AI Collaboration 1 Flatiron Institute 2 Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM 3 New York University 4 Princeton University · 2026

Read Representation Learning for Spatiotemporal Physical Systems by Helen Qu 1 Rudy Morel 1 Michael McCabe 1,4 Alberto Bietti 1 François Lanusse 2 Shirley Ho 1,3,4 Yann LeCun 3 The Polymathic AI Collaboration 1 Flatiron Institute 2 Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM 3 New York University 4 Princeton University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13227.txt.

Introduction

Helen Qu and co-authors test how well self-supervised methods learn physics-grounded representations. Most work on spatiotemporal physical systems aims at next-frame prediction, which is costly and error-prone. The team instead judges models on downstream tasks, like estimating governing physical parameters. Generic self-supervised methods sometimes beat methods built for physical modeling, they find. Methods that learn in latent space, such as JEPAs, beat pixel-level prediction methods. Parameter estimates thus offer a clear window on what a model has truly learned.

Catalog record

CREATORHelen Qu 1 Rudy Morel 1 Michael McCabe 1,4 Alberto Bietti 1 François Lanusse 2 Shirley Ho 1,3,4 Yann LeCun 3 The Polymathic AI Collaboration 1 Flatiron Institute 2 Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM 3 New York University 4 Princeton University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18173
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13227

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Representation Learning for Spatiotemporal Physical Systems Helen Qu 1 Rudy Morel 1 Michael McCabe 1,4 Alberto Bietti 1 François Lanusse 2 Shirley Ho 1,3,4 Yann LeCun 3 The Polymathic AI Collaboration 1 Flatiron Institute 2 Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM 3 New York University 4 Princeton University Abstract Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system’s evolution in time. However, these emulators are computationally expensive to train and are subject to performance pitfalls, such as compounding errors during autoregressive rollout. In this work, we take a different perspective and look at scientific tasks further downstream of predicting the next frame, such as estimation of a system’s governing physical parameters. Accuracy on these tasks offers a uniquely quantifiable glimpse into the physical relevance of the representations of these models. We evaluate the effectiveness of general-purpose self-supervised methods in learning physics-grounded representations that are useful for downstream scientific tasks. Surprisingly, we find that not all methods designed for physical modeling outperform generic self-supervised learning methods on these tasks, and methods that learn in the latent space (e.g., joint embedding predictive architectures, or JEPAs) outperform those optimizing pixel-level prediction objectives. † † † Code: https://github.com/helenqu/physical-representation-learning . 1 Introduction Understanding and forecasting of physical systems is a challenging problem with applications ranging from biological development to astrophysical phenomena ( 20 ; 26 ) . Many applications of machine learning in this space explore autoregressive surrogate modeling, which aims to learn frame-by-frame, pixel-by-pixel emulators of computationally expensive numerical simulations ( 21 ; 17 ; 23 , e.g.,) . However, these full-field prediction models are computationally expensive to train and may not be best suited for higher-level downstream tasks of scientific interest, such as parameter estimation or qualitative prediction (e.g., whether the system remains laminar or becomes turbulent). Relatively little attention has been paid to understanding which learning paradigms optimally learn and preserve physically meaningful information. In this work, we investigate the efficacy of self-supervised learning paradigms for tackling scientifically meaningful tasks in spatiotemporal physical systems. We compare traditional masked autoencoding and joint embedding predictive architectures ( 18 ; 1 ; 5 ; 2 , JEPAs,) with methods developed for physical modeling and inference on three representative physical systems (active matter, shear flow, and Rayleigh-Bénard convection; see Figure 1 ). Unlike traditional self-supervised methods, JEPAs are trained to predict in the model’s learned latent space rather than the granular, low-level space of pixel values. We probe the models’ understanding of these systems through physical parameter estimation, a quantifiable proxy for physical information, and demonstrate that embeddings learned by latent prediction models consistently outperform pixel-based self-supervised methods. Figure 1: Example trajectories from the physical systems in our evaluation. 2 Setup We introduce the self-supervised representation learning methods, physical modeling baselines, and fine-tuning procedure used in this work. 2.1 Representation Learning Frameworks Joint Embedding Predictive Architectures. We introduce a latent feature prediction model based on joint embedding predictive architectures (JEPA) optimized for dynamics. JEPAs minimize error in the space of representations, while pixel reconstruction models learn low-level detail in the visual input that may not be helpful for understanding the simple underlying dynamics. Our JEPA formulation posits that for a sample with T temporal steps x_{0:T} , aligning the representation of x_{t:t+k} with the representation of x_{t+k:t+2k} (i.e. predicting the representation of the next sequence of k frames from a context sequence of k frames) will produce flexible embeddings that carry high-level information about the visual content. Formally, we aim to learn an encoder f:\mathcal{X}\rightarrow\mathcal{Z} and predictor g:\mathcal{Z}\rightarrow\mathcal{Z} that minimizes \mathcal{L}(f,g)=\mathbb{E}_{x_{i},x_{i+1}\sim\mathcal{X}}[\ell_{\text{VICReg}}(g(f(x_{i})),f(x_{i+1})] , where \ell_{\text{VICReg}} is defined following 6 to prevent mode collapse: \displaystyle\ell_{\text{VICReg}}(z_{i},z_{i+1})=\lambda s(z_{i},z_{i+1})+\mu[v(z_{i})+v(z_{i+1})]+\nu[c(z_{i})+c(z_{i+1})] (1) Here, s(z_{i},z_{i+1})=\frac{1}{n}\sum_{i}\left\lVert z_{i+1}-z_{i}\right\rVert^{2}_{2} is the invariance criterion, v(z)=\frac{1}{d}\sum_{j=1}^{d}\text{max}(0,1-\sqrt{\text{var}(z)+\epsilon}) is the variance regularization term, and c(z)=\frac{1}{d}\sum_{i\neq j}C_{ij}(z)^{2} is the covariance regularization term, where C is the covariance matrix of z . \lambda,\mu,\nu are hyperparameters controlling the importance of each term in the loss, and d is the batch size. Masked autoencoding. Autoencoders aim to learn an encoder-decoder pair f:\mathcal{X}\rightarrow\mathcal{Z},g:\mathcal{Z}\rightarrow\mathcal{X} , where \mathcal{Z} is the abstract representation space of the model, that minimizes pixel-level reconstruction error. We implement masked autoencoding in this work, where reconstruction error is computed over masked regions x(m),m\in\{0,1\}^{N} : \mathcal{L}(f,g)=\mathbb{E}_{x\sim\mathcal{X},m\sim\mathcal{M}}[\hat{x}(m)-x(m))^{2}] , where \hat{x}=g(f(x)) . In practice, masked autoencoders have been extended to spatiotemporal data by enforcing temporal tube masking, i.e. all frames x_{0:T} use the same spatial mask m . Autoencoding is a common paradigm for feature learning from large datasets. 2.2 Physical Modeling Baselines Our baselines center their approaches on learning physical priors, either through data (in the style of autoregressive foundation models) or through architectural/training inductive biases (neural operator-style approaches). Autoregressive foundation models. Foundation models for physics, often implemented as pixel-level autoregressive models for spatiotemporal systems, learn to predict the pixel values of the frame at the next timestep x_{t+1} given a context window of the previous n frames, x_{t-n:t} . These are often called “surrogate models” due to their potential to replace the computationally expensive procedure of numerically solving for the next step of the spatiotemporal system (e.g., through PDE numerical solvers). In-context operator learning. In-context operator learning models combine in-context transformers with the inductive bias of neural operators. Rather than learning a single operator across all trajectories, that are described by multiple physics, this approach infers a trajectory-specific operator network f_{\theta} , that is, an evolution rule, from a short context window and evolves it forward in time using explicit integration. 2.3 Fine-tuning Fine-tuning learns a prediction head h:\mathcal{Z}\rightarrow\mathcal{Y} on embeddings from the pretrained model to optimize \mathcal{L}(h)=\mathbb{E}_{x\sim\mathcal{X},y\sim p^{*}(\cdot\mid x)}[\ell_{\text{ft}}(h(f(x)),y)] , where \ell_{\text{ft}}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R} is implemented as squared error loss for regression. 3 Evaluating Representations with Physical Systems We evaluate on the task of physical parameter prediction: the minimum prediction error achievable on physical parameters underlying each system. Intuitively, these parameters govern the time evolution of these systems, so low inference error indicates better understanding of the underlying physical process. We show some examples of how evolution depends on the values of these parameters in Figure 2 . We evaluate across three PDE-governed spatiotemporal systems drawn from The Well ( 28 ) . Active matter. Active matter systems are a collection of agents that convert chemical energy into mechanical work, causing emergent, system-wide patterns and collective dynamics. This dataset models the dynamics of N rodlike active particles immersed in a Stokes fluid, which is modeled by kinetic theory. The system parameters of interest are \alpha , the active dipole strength, and \zeta , the strength of particle alignment through steric interactions. Rayleigh-Bénard convection. This system describes the behavior of a horizontal fluid layer heated from below and cooled from above, forming Bénard convective cells due to the temperature gradient. The system parameters of interest parameterize properties of the fluid layer: the Rayleigh number \nu , the ratio of buoyancy forces to viscous forces, and the Prandtl number \kappa , the ratio of momentum diffusivity to thermal diffusivity. Shear flow. Shear flow describes the boundary between layers of fluid (modeled by incompressible Navier-Stokes) moving parallel to each other at different velocities, potentially leading to vortex/eddy formation and turbulence. The system parameters of interest are the Reynolds number, the ratio between

Read the complete text