Optimizing Data Distribution and Kernel Performance for Efficient Training of Chemistry Foundation Models: A Case Study with MACE
Jesun Firoz, Franco Pellegrini, Mario Geiger, Darren Hsu, Jenna A. Bilbrey, Han-Yi Chou, Maximilian Stadler, Markus Hoehnerbach, Tingyu Wang, Dejun Lin, Emine Kucukbenli, Henry W. Sprueill, Ilyes Batatia, Sotiris S. Xantheas, MalSoon Lee, Chris Mundy, Gabor Csanyi, Justin S. Smith, Ponnuswamy Sadaya · 2025
Read Optimizing Data Distribution and Kernel Performance for Efficient Training of Chemistry Foundation Models: A Case Study with MACE by Jesun Firoz, Franco Pellegrini, Mario Geiger, Darren Hsu, Jenna A. Bilbrey, Han-Yi Chou, Maximilian Stadler, Markus Hoehnerbach, Tingyu Wang, Dejun Lin, Emine Kucukbenli, Henry W. Sprueill, Ilyes Batatia, Sotiris S. Xantheas, MalSoon Lee, Chris Mundy, Gabor Csanyi, Justin S. Smith, Ponnuswamy Sadaya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10700.txt.
Introduction
Jesun Firoz and co-authors present a 2025 study on training chemistry foundation models faster and more cheaply. Such models learn from three-dimensional maps of molecules. Unlike models fed one huge graph, these chew through many small graphs of mixed sizes.
That mix demands its own tricks for splitting data across machines and tuning the math kernels. The team uses one model family, MACE, as its test case. Their work targets both how data is spread out and how fast each kernel runs. Faster training means chemists can test more ideas about new molecules and materials. The lessons should transfer to other models built on the same graph style.
Catalog record
| CREATOR | Jesun Firoz, Franco Pellegrini, Mario Geiger, Darren Hsu, Jenna A. Bilbrey, Han-Yi Chou, Maximilian Stadler, Markus Hoehnerbach, Tingyu Wang, Dejun Lin, Emine Kucukbenli, Henry W. Sprueill, Ilyes Batatia, Sotiris S. Xantheas, MalSoon Lee, Chris Mundy, Gabor Csanyi, Justin S. Smith, Ponnuswamy Sadaya |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Data Distribution and Kernel Performance for Efficient Training of Chemistry Foundation Models: A Case Study with MACE Jesun Firoz, Franco Pellegrini, Mario Geiger, Darren Hsu, Jenna A. Bilbrey, Han-Yi Chou, Maximilian Stadler, Markus Hoehnerbach, Tingyu Wang, Dejun Lin, Emine Kucukbenli, Henry W. Sprueill, Ilyes Batatia, Sotiris S. Xantheas, MalSoon Lee, Chris Mundy, Gabor Csanyi, Justin S. Smith, Ponnuswamy Sadayappan, Sutanay Choudhury Abstract . Chemistry Foundation Models (CFMs) that leverage Graph Neural Networks (GNNs) operating on 3D molecular graph structures are becoming indispensable tools for computational chemists and materials scientists. These models facilitate the understanding of matter and the discovery of new molecules and materials. In contrast to GNNs operating on a large homogeneous graphs, GNNs used by CFMs process a large number of geometric graphs of varying sizes, requiring different optimization strategies than those developed for large homogeneous GNNs. This paper presents optimizations for two critical phases of CFM training: data distribution and model training, targeting MACE - a state-of-the-art CFM. We address the challenge of load balancing in data distribution by formulating it as a multi-objective bin packing problem. We propose an iterative algorithm that provides a highly effective, fast, and practical solution, ensuring efficient data distribution. For the training phase, we identify symmetric tensor contraction as the key computational kernel in MACE and optimize this kernel to improve the overall performance. Our combined approach of balanced data distribution and kernel optimization significantly enhances the training process of MACE. Experimental results demonstrate a substantial speedup, reducing per-epoch execution time for training from 12 to 2 minutes on 740 GPUs with a 2.6M sample dataset. 1. Introduction Atomistic simulation serves as the cornerstone of modern computational chemistry and materials science. For these applications, highly accurate quantitative calculations of molecular and material properties are essential. Density Functional Theory (DFT), a quantum mechanical ab-initio (or first-principles) approach, has long been the preferred method for such calculations due to its relatively high generality and accuracy. However, the high time complexity and significant computational resources needed for DFT calculations limit simulations to nanosecond timescales and nanometer length scales ( Kulichenko et al., 2021 ; Pan, 2021 ) . Recently, machine learning interatomic potentials (MLIPs), computational models to map atomic environments to energies and forces, have emerged as attractive alternatives to DFT calculations ( Kulichenko et al., 2021 ; Ko and Ong, 2023 ; Wang et al., 2024 ) . The time-to-solution of these surrogate models is significantly shorter while their accuracy is comparable to that of DFT ( Käser et al., 2023 ) . Figure 1 . Key contrast between different classes of GNNs: Molecular GNNs are trained on many small graphs instead of a single big graph. While traditional specialized surrogate models excel at predicting properties for specific chemical systems, the recently-emerging foundation models are demonstrating broad capabilities in predicting atomistic properties such as potential energy, forces, stress, and charges for a diverse set of chemical species ( Bilbrey et al., 2020 ; Batatia et al., 2023 ; Allen et al., 2024 ; Shoghi et al., 2024 ; Pasini et al., 2024 ; Zubatyuk et al., 2021 ) . Foundation models, trained on large and diverse datasets simultaneously , are capable of performing a wide range of general tasks ( Bommasani et al., 2021 ) . The objective of a chemistry foundation model (CFM) is to unify training over a large collection of molecular and materials datasets to improve zero-shot accuracy and facilitate fine-tuning on cases with limited data. When used in zero-shot setting, CFMs can handle a wider range of chemical species than specialized models. During fine-tuning, CFMs reduce the need for extensive data collection and retraining for each new task by leveraging previously learned knowledge from diverse chemical domains ( Niblett et al., 2024 ) . Graph Neural Networks (GNNs) are the dominant architecture for chemistry foundation models (CFMs), with equivariant GNNs favored for their ability to preserve rotational symmetries in atomic systems ( Satorras et al., 2021 ; Batzner et al., 2022 ) . However, efficiently capturing many-body interactions (meaning how many neighboring atoms/vertices are considered simultaneously to compute contribution to per atom energy) —a critical aspect of modeling chemical systems—remains a challenge due to the high computational costs of operations on higher-order tensors. To address this, the atomic cluster expansion (ACE) formalism ( Drautz, 2019 ) was developed to describe many-body interactions efficiently. Building on ACE, Batatia et al. ( Batatia et al., 2022 ) introduced MACE, which integrates these interactions into equivariant message-passing schemes, improving computational efficiency without sacrificing accuracy. MACE extends beyond pairwise interactions to capture complex multi-atom effects and has been widely adopted as the state-of-the-art CFM for molecular and materials modeling ( Batatia et al., 2022 ; Batatia et al., 2023 ; Kovács et al., 2023 ; Batatia et al., 2025 ) . Key Traits of CFM Training Workloads. CFMs present distinctly different scalability challenges compared to conventional GNN workloads (cf. Figure 1 and Table 1 ), which typically focus on single massive graph ( Gong and Kumar, 2024 ; Li et al., 2024 ; Xia et al., 2024 ) . Instead, CFMs train on collections of small, irregular 3D molecular graphs, requiring careful data distribution to avoid load imbalance and straggler effects across GPUs. These molecular graphs vary in size, sparsity, and chemical composition, making efficient batching and workload partitioning a nontrivial problem. Another fundamental trait of CFM workloads is the dominance of symmetric tensor contractions—high-cost operations central to equivariant GNNs. These tensor operations scale poorly without specialized optimizations, making them a primary bottleneck in large-scale training. Given that CFM training can require anywhere from \approx 3000 ( Batatia et al., 2023 ) to \approx 34000 GPU-hours ( Shoghi et al., 2024 ) , optimizing tensor computations and load balancing is critical for scalability. Why is MACE a representative CFM workload? Many state-of-the-art equivariant GNNs, including MACE and NequIP ( Batzner et al., 2022 ) , share a common mathematical foundation, where different models correspond to specific parameter choices within a unified design space ( Batatia et al., 2025 ) . This commonality highlights shared computational challenges, particularly the high cost of symmetric tensor contractions, making MACE an ideal target for optimization. Beyond its architectural significance, MACE has demonstrated remarkable generalization across diverse chemical systems. Unlike traditional machine-learned force fields that require system-specific training, MACE has been successfully applied to a broad range of atomistic simulations, including solids, liquids, gases, chemical reactions, and interfaces ( Batatia et al., 2023 ; Batatia et al., 2022 ; Batatia et al., 2025 ) . This broad applicability positions MACE as a strong candidate for a foundation model in chemistry, reinforcing the importance of optimizing its computational efficiency. Therefore, by focusing on MACE, we ensure that our optimizations extend to a wider class of CFMs built on equivariant GNNs, making them broadly impactful for large-scale atomistic modeling. Aspect GNNs for large Graphs GNNs for Geometric/Molecular Graphs Input graph structure Single massive graph Collection of many smaller graphs Node count Millions to billions Typically <1000 per graph Data distribution Usually a single connected graph that needs to be partitioned across workers (partition vertices and edges) Natural partitioning as each molecular graph is independent. Each worker processes multiple graphs