Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows

Janne Perini Rafael Bischof Moab Arar Ayça Duran, Michael A. Kraus Siddhartha Mishra Bernd Bickel · 2026

Read Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows by Janne Perini Rafael Bischof Moab Arar Ayça Duran, Michael A. Kraus Siddhartha Mishra Bernd Bickel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21210.txt.

Introduction

Perini and co-authors turn a pretrained video model into a fast wind simulator for city design. Pedestrian wind comfort and safety need time resolved fluid dynamics runs, which are too costly for wide design search. Their WinDiNet fine-tunes a two billion parameter video diffusion model on ten thousand 2D flow simulations over generated building layouts. It produces full rollouts in under a second. Since the model is differentiable, building positions can be tuned by gradient descent for wind goals. Ground truth simulations confirm the improved layouts.

Catalog record

CREATORJanne Perini Rafael Bischof Moab Arar Ayça Duran, Michael A. Kraus Siddhartha Mishra Bernd Bickel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66042
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows Janne Perini Rafael Bischof Moab Arar Ayça Duran, Michael A. Kraus Siddhartha Mishra Bernd Bickel Abstract Designing urban spaces that provide pedestrian wind comfort and safety requires time-resolved Computational Fluid Dynamics (CFD) simulations, but their current computational cost makes extensive design exploration impractical. We introduce WinDiNet (Wind Diffusion Network), a pretrained video diffusion model that is repurposed as a fast, differentiable surrogate for this task. Starting from LTX-Video, a 2B-parameter latent video transformer, we fine-tune on 10,000 2D incompressible CFD simulations over procedurally generated building layouts. A systematic study of training regimes, conditioning mechanisms, and VAE adaptation strategies, including a physics-informed decoder loss, identifies a configuration that outperforms purpose-built neural PDE solvers. The resulting model generates full 112-frame rollouts in under a second. As the surrogate is end-to-end differentiable, it doubles as a physics simulator for gradient-based inverse optimization: given an urban footprint layout, we optimize building positions directly through backpropagation to improve wind safety as well as pedestrian wind comfort. Experiments on single- and multi-inlet layouts show that the optimizer discovers effective layouts even under challenging multi-objective configurations, with all improvements confirmed by ground-truth CFD simulations. 1 Introduction When urban designers and engineers plan modern, livable public spaces, wind is a key constraint. It affects pedestrian comfort, structural loads on buildings, and natural ventilation. Buildings that channel airflow can produce gusts that endanger pedestrians and stress building facades, while overly sheltered configurations create stagnant zones where heat and pollutants accumulate. In cities increasingly affected by air pollution and rising temperatures, these aspects carry real consequences for structural safety, public health, and quality of life. Exploring urban layouts that navigate this trade-off requires fast, accurate flow predictions and, ideally, guidance on how to update a given design to improve wind comfort and safety simultaneously. Computational fluid dynamics (CFD) simulations, the traditional method for this task, are computationally expensive and require tedious modelling and domain expertise in addition to taking minutes to hours at the resolution needed for a reliable comfort evaluation [ 31 ] . Furthermore, they are typically non-differentiable, meaning they offer no signal about which geometric changes would improve a given design. Deep-learning surrogates mitigate these limitations by offering fast inference and differentiable predictions. While existing approaches based on convolutional and graph neural networks can predict mean velocity or wind-factor fields [ 32 , 27 ] , temporally averaged outputs cannot evaluate comfort criteria defined as exceedance probabilities over threshold speeds [ 21 ] , nor capture transient gusts relevant to pedestrian safety. Temporal extensions based on Fourier neural operators and autoregressive transformers [ 39 , 3 ] can in principle produce time-resolved predictions, but they struggle to maintain accuracy over long rollouts when trained from scratch on limited domain-specific data. Meanwhile, video diffusion models have advanced rapidly in capabilities relevant to wind flow prediction. Recent latent diffusion transformers generate high-resolution, temporally coherent sequences with multi-scale motion and long-range spatial dependencies [ 12 , 54 , 1 ] . These models already encode physical priors, such as gravity, collisions, and fluid-like motion, that can be adapted to generate visually plausible physical phenomena [ 56 , 9 , 59 , 55 ] . Predicting urban wind flows fits a similar framing: projected onto a horizontal plane, a velocity field evolves as a frame sequence whose channels encode physical quantities rather than pixel colors. We show that fine-tuning a pretrained video model on synthetic urban CFD data yields a differentiable surrogate accurate enough for both forward prediction and gradient-based inverse optimization of building layouts for pedestrian wind comfort, and that the physical priors transfer to quantitatively accurate simulations, not just visually plausible ones. In this work, we fine-tune LTX-Video [ 12 ] , a transformer-based latent video diffusion model, on a dataset of 13,000 2D CFD simulations over procedurally generated urban footprint layouts and systematically study the design choices needed to obtain physically accurate predictions from a pretrained video backbone (Figure 1 ). We compare (i) the training regime, such as Low-Rank Adaptation (LoRA) [ 15 ] vs . full fine-tuning, (ii) the conditioning mechanism (text prompts vs . scalar-conditioned latent modulation with the text encoder removed), and (iii) the Variational Autoencoder (VAE) adaptation strategy, including color adapters, decoder fine-tuning, and physics-informed objectives. Our main contributions are: • We generate a dataset of 13,000 2D incompressible wind flow simulations over procedurally generated urban layouts, spanning diverse building configurations, inlet speeds, and domain sizes. The dataset, code and model weights will be released upon publication. • We present a systematic study of adapting a pretrained video diffusion model for physics simulation, comparing training regimes, conditioning mechanisms, and VAE adaptation strategies, including physics-informed decoder losses. • We demonstrate that the resulting differentiable surrogate enables gradient-based inverse optimization of building layouts for pedestrian wind comfort. Figure 1 : Overview of the proposed framework. (a) Procedurally generated urban layouts are simulated with a 2D incompressible Euler solver to produce training data. (b) A latent diffusion model with a physics-informed VAE is trained to generate wind field sequences conditioned on building footprint, inlet speed u_{\mathrm{in}} , and domain size L . (c) At inference, the model generates horizontal and vertical velocity fields (u,v) and enables gradient-based inverse optimization of building layouts. 2 Related Work Pedestrian wind comfort depends on transient flow phenomena (vortex shedding, recirculation zones, shear-layer instabilities) that are computationally expensive to resolve numerically [ 31 ] . A large body of work has therefore pursued data-driven surrogates trained on CFD datasets. Convolutional neural networks (CNNs) and CNN–generative adversarial network (GAN) hybrids map building footprints to mean velocity or wind-factor fields [ 32 , 30 , 46 , 6 , 19 , 5 ] . Graph neural networks operate on unstructured meshes at city scale [ 27 , 10 ] , including physics-informed variants that embed Reynolds-Averaged Navier–Stokes (RANS) residuals in the loss [ 44 ] . Physics-informed neural networks have also been applied to reconstruct 3D wind fields around buildings from sparse measurements [ 41 ] , demonstrating that embedding governing equations in the training loss can compensate for limited data. These approaches predict temporally averaged or steady-state conditions. However, the wind comfort criteria applied in practice [ 21 , 17 ] are formulated as exceedance probabilities over threshold speeds, which require time-resolved velocity data to evaluate. Although Fourier Neural Operator (FNO) [ 25 ] variants [ 39 , 3 ] can predict instantaneous states and generate temporal sequences through autoregressive rollout, they struggle with generating long rollouts because errors accumulate over successive steps. Three recent threads suggest how to close this gap. First, diffusion models have been shown to generate accurate time-dependent physical fields [ 7 , 33 ] . Second, partial differential equation (PDE) solving has been recast as video generation [ 23 ] . Third, foundation models pretrained on diverse equation families [ 13 , 34 , 47 , 57 ] , enabled by large-scale PDE benchmarks [ 48 , 36 ] , improve sample efficiency on downstream scientific tasks. These developments motivate formulating the unsteady urban wind flow simulation as a conditional video generation problem in which temporal coherence and long-range spatial dependencies are learned from the domain-specific

Read the complete text