Jigsaw: Training Multi-Billion-Parameter AI Weather Models With Optimized Model Parallelism Conference: Technical Report; June 25; 2025 CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Distributed algorithms CCS: Computing methodologies Neural networks CCS: Applied c

Deifilia Kieckhefen, Markus Götz, Lars H. Heyen, Achim Streit, Charlotte Debus · 2025

Read Jigsaw: Training Multi-Billion-Parameter AI Weather Models With Optimized Model Parallelism Conference: Technical Report; June 25; 2025 CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Distributed algorithms CCS: Computing methodologies Neural networks CCS: Applied c by Deifilia Kieckhefen, Markus Götz, Lars H. Heyen, Achim Streit, Charlotte Debus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05753.txt.

Introduction

Deifilia Kieckhefen and four co-authors present a training scheme for huge AI weather models in a 2025 technical report. Weather models that learn from data now rival classic methods, but high detail and long lead times demand giant networks and data samples. Memory and data flow then become the bottlenecks on big machines.

Their architecture scales its work in a straight line with input size. The training scheme, called Jigsaw, splits both the map domain and the network tensors across machines, removing repeated memory use. Scaling to hundreds of graphics processors beat prior methods and reached a large share of peak machine speed.

Catalog record

CREATORDeifilia Kieckhefen, Markus Götz, Lars H. Heyen, Achim Streit, Charlotte Debus
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05753

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Jigsaw: Training Multi-Billion-Parameter AI Weather Models With Optimized Model Parallelism Conference: Technical Report; June 25; 2025 CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Distributed algorithms CCS: Computing methodologies Neural networks CCS: Applied computing Earth and atmospheric sciences Deifilia Kieckhefen, Markus Götz, Lars H. Heyen, Achim Streit, Charlotte Debus Abstract . AI-based methods have revolutionized atmospheric forecasting, with recent successes in medium-range forecasting spurring the development of climate foundation models. Accurate modeling of complex atmospheric dynamics at high spatial resolutions and longer lead times requires large neural networks and gigabyte-sized data samples, making accelerator memory and I/O-bandwidth the bottlenecks for model training. We introduce WeatherMixer, a multi-layer-perceptron-based architecture whose workload scales linearly with input size, allowing the model to learn global weather phenomena at accuracies similar to numerical weather prediction. To cope with the computational demand, we propose Jigsaw, a novel model parallelization scheme that employs both domain and tensor parallelism, eliminating memory redundancy. Jigsaw exceeds state-of-the-art performance in strong scaling in compute–communication-limited systems and achieves superscalar weak scaling in I/O-bandwidth-limited systems. We scale training to 256 GPUs, reaching peak performances of 9 and 11 PFLOPs—23% and 28% of theoretical peaks—achieving 68% and 72% scaling efficiency versus 51% without model parallelism. 1-way Longitude Latitude Var. 2-way 4-way W W_{1} W_{2} W_{1} W_{2} W_{3} W_{4} Domain-parallel across variables Domain-parallel across space Tensor-parallel Tensor-parallel Encoder Token-mixing Channel-mixing Decoder Problem : Transformers scale poorly Problem : Atmospheric data are gigabytes large Problem : Model sizes limited by GPU memory Solution : Scales linearly; learns global patterns Solution : Parallel data loading Solution : Partition data, model, optimizer WeatherMixer Jigsaw Parallelism Performance : 11 PFLOPs on 256 GPUs I/O-Bandwidth-limited : superscalar weak scaling Compute–comm.-limited : exceeds SOTA strong scaling Visual depiction of Jigsaw parallelism scheme and WeatherMixer. 1. Introduction The recent emergence of accurate, artificial intelligence (AI)-based atmospheric forecast models is a revolution in meteorology. Historically, this discipline has been dominated by process-based numerical modeling. Practically, this entails solving partial differential equations that describe atmospheric dynamics with numerical methods. Data-driven models instead learn purely from historic data without requiring explicit formulations of the underlying physical equations. The predictive skill of data-driven models now exceeds that of classical numerical approaches in many of the standard forecast scores ( Keisler 2022 ; Lam et al. 2023 ; Bi et al. 2023 ; Nguyen et al. 2024 ) . Almost all of the recent breakthroughs in data-driven weather forecasting and initial attempts towards atmospheric foundation models are based on the Transformer neural architecture ( Vaswani et al. 2017 ) . The predictive capabilities of the attention mechanism have pushed data-driven models from an interesting research question towards the possibility of operational deployment. Current attempts at further improving forecasting skill ( Wang et al. 2024 ; Bodnar et al. 2024 ) generally try to exploit neural network scaling laws ( Kaplan et al. 2020 ) , i.e., an increase in the number of free trainable parameters in the model and a larger training dataset size should result in better generalizability and predictive performance. However, the resulting models have immense sizes. In combination with the large size of the input data, the memory capacity of individual accelerators (GPUs) is rapidly exceeded. To overcome this, different parallelization and distributed-memory computation approaches have been pursued. Next to the widely employed data parallelism, i.e., distributing the dataset into disjoint batches across computation processes and training local model copies on these data chunks, more recent works have turned to model parallelism (MP), i.e., distributing the weights of the models themselves. Approaches like Megatron-LM ( Shoeybi et al. 2020 ) , ZeRO ( Rajbhandari et al. 2020 ) , and FSDP ( Zhao et al. 2023 ) , originally developed for large language models (LLMs), do not tackle scalability limitations of Transformers with respect to global atmospheric data well. For one, the individual input samples in atmospheric sciences are orders of magnitude larger in size than that of LLMs, due to the highly-resolved image-like data with tens or even hundreds of individual state variables (channels). Transformer-based atmospheric models commonly employ tokenization in the spatial domain, thus suffering from a quadratic computational complexity of the self-attention mechanism. While current models are still able to handle the global 0.25° resolution of the commonly used ECMWF Reanalysis v5 dataset (ERA5) ( Hersbach et al. 2020 ) by training with only one or two data items per accelerator at a time, higher spatial resolutions, as to be expected for ERA6 or CMIP6, will inevitably exceed the hardware capabilities. Second, for true operational deployment, high predictive performance for longer lead times in the three- to seven-day medium-range forecast time is required. In this temporal range, atmospheric phenomena at global spatial scales become important. Thus, “global vision” of the model is necessary—this is currently infeasible for current modeling approaches and operational compute budgets. Current Transformer-based models are limited to computing highly localized spatial extents, e.g., via shifted window attention ( Liu et al. 2021 ) . Similarly, to achieve a sufficient spatial interaction range in graph-based neural models, the network’s depth needs to be increased linearly. This inherently implies increasing model size and, correspondingly, computational load. In response to these challenges for large data-driven meteorological models—higher spatial resolutions and longer rollout times—we propose WeatherMixer, an atmospheric model based on the MLP-Mixer architecture ( Tolstikhin et al. 2021 ) . Multi-layer perceptions (MLPs) have experienced a revival, as a number of studies demonstrated their competitive predictive skills for time-series forecasting ( Ekambaram et al. 2023 ; Chen et al. 2023a ) . From a computational perspective, this has the major advantage that such feed-forward models reduce to a sequence of matrix–matrix multiplications, as opposed to the complex computational pathway of the self-attention mechanism. As such, WeatherMixer provides a framework for model parallelism across all network layers via distributed dense matrix–matrix multiplications. In this work, we extend upon this principle with Jigsaw parallelism, which provides efficient and scalable intra-node distribution of the WeatherMixer towards multi-billion-parameter model capacities. Our main contributions are WeatherMixer, a new MLP-based architecture for atmospheric dynamics and climate forecasting; and Jigsaw, a novel parallelism approach that fully shards data, model, and optimizer states. Jigsaw partitions model weights and training data across one dimension when distributed over two compute devices and two dimensions when distributed over four compute devices within a node. For the first time in a data-driven atmospheric model, we present roofline plots that show the I/O- and computation–communication-bounds when working with different precisions. We present strong scaling results that rival state-of-the-art LLM-parallelism approaches and show that Jigsaw achieves superscalar weak scaling in I/O-bandwidth-limited systems. To quantify the resources consumed for training, we also report the measured energy consumption and estimate carbon-emissions footprint for our primary training experiments. 2. Related Work 2.1. Data-driven atmospheric models Although purely data-driven atmospheric models have been explored for some time ( Kurth et al. 2018 ; Dueben and Bauer 2018 ; Rasp and Thuerey 2021 ) , only recent developments, occurring over the past three years, have lead to fundamental breakthroughs. Among the first models to be published was FourCastNet ( Kurth et al. 2023a ) . Utilizing adaptive Fourier neural operators, it is able to predict twenty atmospheric state variables at

Read the complete text