MSPT : Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention

Pedro M. P. Curvo, Jan-Willem van de Meent, Maksim Zhdanov · 2025

Read MSPT : Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention by Pedro M. P. Curvo, Jan-Willem van de Meent, Maksim Zhdanov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01738.txt.

Introduction

Pedro Curvo, Jan-Willem van de Meent, and Maksim Zhdanov scale neural solvers to industrial physics. Large simulations must capture fine local acts and long-range links at once. Millions of spatial elements make both hard. Attention over all elements is too costly.

Their Multi-Scale Patch Transformer splits the task. Local attention works inside patches of elements. Global attention works on coarse patch summaries. That split keeps fine detail without full cost. Industrial users could run larger, truer physics models with learned speed. Patch summaries carry the far field cheaply, much as multigrid solvers do, which is why the design scales near linearly in tests.

Catalog record

CREATORPedro M. P. Curvo, Jan-Willem van de Meent, Maksim Zhdanov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01738

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MSPT : Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention Pedro M. P. Curvo, Jan-Willem van de Meent, Maksim Zhdanov Abstract A key scalability challenge in neural solvers for industrial-scale physics simulations is efficiently capturing both fine-grained local interactions and long-range global dependencies across millions of spatial elements. We introduce the Multi-Scale Patch Transformer (MSPT), an architecture that combines local point attention within patches with global attention to coarse patch-level representations. To partition the input domain into spatially-coherent patches, we employ ball trees, which handle irregular geometries efficiently. This dual-scale design enables MSPT to scale to millions of points on a single GPU. We validate our method on standard PDE benchmarks (elasticity, plasticity, fluid dynamics, porous flow) and large-scale aerodynamic datasets (ShapeNet-Car, Ahmed-ML), achieving state-of-the-art accuracy with substantially lower memory footprint and computational cost. 1 Introduction Deep learning is increasingly applied to model complex physical phenomena as a computationally efficient surrogate. However, as applications in computational fluid dynamics (CFD) and multi-physics design scale to simulations with millions of mesh points, these methods face substantial computational challenges 4 . Developing neural architectures capable of efficiently processing millions of spatial elements has therefore become important 4 , particularly for high-throughput environments such as large-scale design optimization and real-time industrial analysis where both accuracy and scalability are required. These challenges require neural surrogates that capture fine-grained physical interactions while maintaining computational efficiency. Peak Memory (GB) 42.8 26.0 Latency (ms) 31 28 Pool Transolver MSPT Figure 1 : Parallelized Multi-Scale Attention mechanism. Each patch performs local self-attention, while pooled supernodes exchange information globally across patches in parallel. Peak memory (GB) and latency (ms) on 500k points with 256 slices ( Transolver ) and 256 patches ( MSPT ). A key obstacle in scaling these methods lies in capturing spatial dependencies appropriate to different physical regimes. In solid mechanics, stress and strain fields localize around applied loads. By contrast, in incompressible fluids a global pressure coupling arises from the divergence-free constraint. Similarly, aerodynamics problems require computing forces from surface pressures, which must be consistent with far-field boundary conditions. Modeling such long-range dependencies poses a challenge, since computing all pairwise interactions scales quadratically with the system size. Modeling long-range dependencies at industrial scale therefore requires approximations that preserve accuracy while reducing computational cost ( 33 ) . Recent approaches have made substantial progress toward scalable neural PDE solvers. Neural operator models ( 26 ; 40 ; 16 ) have demonstrated that learning mappings between function spaces enables mesh-independent surrogate models. However, these spectral methods require structured grids or periodic boundary conditions and often struggle to accurately represent sharp, localized features without dense sampling or large frequency bases 47 . Transformer-based solvers 5 ; 18 ; 21 ; 48 extend operator-learning concepts to unstructured data, including meshes and point clouds, by leveraging attention to capture long-range dependencies. A key area of innovation are architectures that avoid quadratic complexity. Transolver 47 projects the input mesh to a reduced set of learnable global representations (slices), achieving sub-linear complexity. However, this approach scales poorly with bottleneck size and relies on pooling that compromises simulation fidelity. UPT 2 introduces a centralized latent space that summarizes the domain via cross-attention between mesh points and shared latent tokens, providing global context applied uniformly across regions. Extending this idea, AB-UPT 1 adopts a branched design for CFD, splitting computation into surface and volume branches that perform self-attention independently and exchange information through cross-attention. Erwin 49 removes pooling entirely and instead achieves linear complexity by localizing attention to ball-tree partitions, preserving local fidelity. However, this approach suffers from limited information propagation in capturing long-range dependencies. In this work, we introduce the Multi-Scale Patch Transformer ( MSPT ), a parallelized multi-scale attention architecture that bridges fine-grained local interactions with global context in a scalable manner. MSPT first partitions the domain into local patches of points, where attention is computed efficiently to capture fine-scale interactions within each region. To capture long-range interactions, we compute a coarse-grained representation of each patch and apply fine-to-coarse attention, allowing information to flow globally without quadratic cost. This multi-scale organization enables MSPT to capture local and global dependencies within a single, unified attention operation, achieving near-linear scaling with the number of points. The main contributions of this work are: • We propose the Parallelized Multi-Scale Attention (PMSA) mechanism, which processes local patch-level interactions and global cross-patch interactions in parallel within a unified attention operation, enabling scalable operator learning with near-linear complexity. • We introduce MSPT , a multi-block transformer architecture that handles arbitrary geometries and varying resolutions through flexible domain partitioning and hierarchical pooling. • We demonstrate state-of-the-art accuracy on standard PDE benchmarks (elasticity, plasticity, fluid dynamics, porous media flow) and industry-scale 3D aerodynamic datasets (ShapeNet-Car, Ahmed-ML) at significantly lower computational cost, potentially scaling to millions of points on a single GPU. 2 Background Neural Operator Learning Neural operators 20 are a class of deep learning models designed to learn mappings between infinite-dimensional function spaces, making them particularly well-suited for parameterizing neural PDE solvers 25 . The key idea is to formulate a model as a sequence of linear integral operators computed in the spectral (Fourier) domain, thus guaranteeing discretization invariance. The main limitations of the original work - specifically suboptimal local feature representations and reliance on uniform grids - were addressed in subsequent works 38 ; 15 ; 28 . In general, spectral methods efficiently capture global dependencies through frequency-domain operations and achieve strong performance on regular, periodic domains. Transformer-based PDE Solvers Complementing spectral approaches, transformer-based architectures leverage attention mechanisms to handle unstructured data such as meshes and point clouds ( 5 ; 29 ; 18 ; 22 ; 21 ; 48 ) . Self-attention naturally models long-range dependencies across irregular domains without requiring regular grid structure. For example, GNOT ( 18 ) augments attention with topology-aware neighborhoods, while OFormer ( 21 ) formulates PDE solution inference as a sequence-to-sequence problem using cross-attention between spatial queries and target fields. CViT 43 leverages recent advancements in computer vision and builds upon a ViT 11 encoder to capture multi-scale dependencies. Overall, transformer-based approaches achieve state-of-the-art performance on most PDE-related tasks. Hierarchical Models To address the scalability bottleneck of attention, hierarchical formulations introduce spatial structure and multi-scale reasoning. ViT 12 first demonstrated the effectiveness of patch-based tokenization in vision, while PatchFormer 8 extended this idea to point clouds by clustering points into compact patch tokens that approximate global interactions. For physical systems, HT-Net 30 constructs explicit coarse-to-fine representations to capture dynamics at multiple scales. Similarly, UPT 2 introduces a latent operator framework where learnable tokens summarize the global domain through cross-attention with physical points, providing compact global context. Erwin 49 imposes regular structure on irregular point cloud data using ball trees, which enables sparse attention and achieves linear complexity. Building upon UPT 2 , AB-UPT 1 introduces a two-branch transformer architecture specifically designed for CFD domains. The model splits computation into a surface branch and a volume branch , allowing each to specialize in distinct physical regimes. Each branch processes its subset of mesh points with self-attention, while cross-attention between branches enables bidirectional information exchange. 3 Multi-Scale Patch Transformer ( MSPT ) We present our Parallelized Multi-Scale Attention mechanism and the overall MSPT architecture

Read the complete text