MATEY: multiscale adaptive foundation models for spatiotemporal physical systems

Pei Zhang, M. Paul Laiu, Matthew Norman, Doug Stefanski, John Gounley · 2024

Read MATEY: multiscale adaptive foundation models for spatiotemporal physical systems by Pei Zhang, M. Paul Laiu, Matthew Norman, Doug Stefanski, John Gounley (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20601.txt.

Description

Abstract Accurate representation of the multiscale features in spatiotemporal physical systems using vision transformer (ViT) architectures requires extremely long, computationally prohibitive token sequences. To address this issue, we propose two adaptive tokenization schemes that dynamically adjust patch sizes based on local features: one ensures convergent behavior to uniform patch refinement, while the other offers better computational efficiency.

Catalog record

CREATORPei Zhang, M. Paul Laiu, Matthew Norman, Doug Stefanski, John Gounley
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MATEY: multiscale adaptive foundation models for spatiotemporal physical systems Pei Zhang, M. Paul Laiu, Matthew Norman, Doug Stefanski, John Gounley Abstract Accurate representation of the multiscale features in spatiotemporal physical systems using vision transformer (ViT) architectures requires extremely long, computationally prohibitive token sequences. To address this issue, we propose two adaptive tokenization schemes that dynamically adjust patch sizes based on local features: one ensures convergent behavior to uniform patch refinement, while the other offers better computational efficiency. Moreover, we present a set of spatiotemporal attention schemes, where the temporal or axial spatial dimensions are decoupled, and evaluate their computational and data efficiencies. We assess the performance of the proposed multiscale adaptive model, MATEY, in a sequence of experiments. The results show that adaptive tokenization schemes achieve improved accuracy without significantly increasing the length of the token sequence. Compared to a full spatiotemporal attention scheme or a scheme that decouples only the temporal dimension, we find that fully decoupled axial attention is less efficient and expressive, requiring more training time and model weights to achieve the same accuracy. Finally, we demonstrate in two fine-tuning tasks featuring different physics that models pretrained on PDEBench data outperform the ones trained from scratch, especially in the low data regime with frozen attention. 1 Introduction Developing foundation models for physical systems is vital for energy generation, earth sciences, and power and propulsion systems. These models offer faster solutions than physics-based simulations and can generalize better across multiple systems than single-purpose AI approaches. However, their application to physical systems, often characterized by multiple sub-processes at different scales, is still in the early stages. For instance, fluid flowing around a cylinder creates a von Kármán vortex street, a highly dynamic flow with rapidly evolving vortices. Accurate solutions of such multiscale systems require a very high resolution representation to capture the most complex features across space and time. However, for scientific machine learning as for modeling and simulation, using very high resolutions to achieve accurate solutions incurs significant computational cost. This is particularly true for developing foundation models using vision transformer (ViT)-based architectures, as using the standard self-attention mechanism for extremely long spatiotemporal sequences can become prohibitively computationally expensive. Efficient representation of multiscale features in high-resolution inputs has been an active research topic in computer vision. Three broad approaches can be characterized. First, multiscale models like Swin Transformer ( Liu et al., 2021 ) and MViTv2 ( Li et al., 2022 ) introduce multiple stages with decreasing resolution and increasing feature dimension for efficient hierarchical representations. Second, computational techniques have been developed that facilitate training on long sequences (e.g., sequence parallelism across GPUs ( Jacobs et al., 2023 ) ) or reduce the effective sequence length in the attention kernel (e.g., decomposing attention along axial directions ( Ho et al., 2019 ) ). Third, the actual sequence length can be directly shortened by pruning and merging tokens ( ( Haurum et al., 2023 ; Meng et al., 2022 ; Yin et al., 2022 ; Bolya & Hoffman, 2023 ) ), though this strategy may lead to critical information loss ( Liu et al., 2024 ) . These techniques have recently been adopted in scientific machine learning (sciML) for physical systems. For example, the atmosphere foundation model Aurora ( Bodnar et al., 2024 ) uses Swin Transformer, while axial attention is applied by MPP ( McCabe et al., 2023 ) . Despite the progress, computational constraints remain a bottleneck, as existing approaches do not yet handle high-fidelity solutions of applications such as computational fluid dynamics, in which input sequences can easily exceed billions of tokens. More efficient algorithms are needed to enable the development of foundation models for multiscale multiphysics systems. In this work, we develop a multiscale adaptive foundation model, MATEY (see Figure 1 ), that provides two key algorithmic contributions to address the challenges posed by spatiotemporal physical systems. First, inspired by the adaptive mesh refinement (AMR) technique, we introduce an adaptive tokenization method that dynamically adjusts patch sizes across the system based on local features, which provides as much as a 2\times reduction in compute for similar or higher accuracy. Second, we present a set of spatiotemporal attention schemes based on the axial attention ( Ho et al., 2019 ) that differ in their decomposition of long spatiotemporal sequences and identify the cost in time-to-accuracy for axial attention. Finally, we assess the fine-tuning performance of models pretrained on PDEBench ( Takamoto et al., 2022 ) in two highly out-of-distribution settings, colliding thermals and magnetohydrodynamics (MHD), that include additional physical variables not included in pretraining and observe the pretrained models outperforming random initialized models. 2 Related work Scientific foundation models. Several research directions have been explored for building foundation models for physical systems, including multiple physics pretraining ( McCabe et al., 2023 ) with PDEBench data, input augmentation with PDE system configurations ( Hang et al., 2024 ) , robust pretraining schemes ( Hao et al., 2024 ) , fine-tuning effectiveness investigations ( Subramanian et al., 2024 ) , and data-efficient multiscale ViT architectures ( Herde et al., 2024 ) . While these work made remarkable progress, they do not directly address the issue of token sequence length, which becomes a computation bottleneck when applying ViTs to high dimension or high resolution data. Multiscale ViTs. While most multiscale ViTs achieve hierarchical representations via multi-stage attention blocks at different resolutions (e.g., MViTv2 ( Li et al., 2022 ) and Swin Transformer ( Liu et al., 2021 ) ), there are a few focusing on tokenization schemes, such as ( Yin et al., 2022 ; Fan et al., 2024 ; Zhang et al., 2024 ; Havtorn et al., 2023 ) . Among these, the single-stage MSViT with dynamic mixed-scale tokenization ( Havtorn et al., 2023 ) , which leverages a learnable gating neural network for selecting the token refinement, is most related to our work. This approach requires a tailored gate loss function and an adaptive trimming scheme to handle the high overhead cost, which in return hurts gate training accuracy. In contrast, the tokenization scheme in MATEY adaptively adjusts the patch sizes directly based on local feature scales, which is simpler and more direct. Axial attentions. The quadratic scaling nature of attention makes it computationally prohibitive for extremely long token sequences from multidimensional systems. To address this challenge, ( Ho et al., 2019 ) proposed the axial attention, which decomposes the full attention into a sequence of attention operations along each axis. It reduces the attention cost from \mathcal{O}(N^{2d}) to \mathcal{O}(N^{d+1}) , for a given d -dimensional system with N^{d} tokens. ViViT ( Arnab et al., 2021 ) factorized the spatiotemporal attention into spatial- and temporal-dimensions for video classification. ( McCabe et al., 2023 ) applied the axial attention in the Axial ViT (AViT) for spatiotemporal solutions of physical systems. While these spatiotemporal attention schemes can reduce the sequence length and hence the attention cost, their impact on accuracy in physical systems is unclear. 3 MATEY, explained We propose multiscale adaptive foundation models, MATEY, to predict two-dimensional spatiotemporal solutions of multiple physical systems. The architecture of MATEY is illustrated in Figure 1 . Given a sequence of T past solutions of some physical system leading up to time

Read the complete text