MegaFold: System-Level Optimizations for Accelerating Protein Structure Prediction Models

Hoa La 1 §, Ahan Gupta §, Alex Morehead, Jianlin Cheng, Minjia Zhang · 2025

Read MegaFold: System-Level Optimizations for Accelerating Protein Structure Prediction Models by Hoa La 1 §, Ahan Gupta §, Alex Morehead, Jianlin Cheng, Minjia Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20686.txt.

Introduction

Hoa La, Ahan Gupta, Alex Morehead, Jianlin Cheng, and Minjia Zhang build MegaFold, a system that makes training protein structure models far cheaper. New models in the AlphaFold line reason over three-dimensional attention maps whose cost grows steeply with chain length, so training them strains graphics card memory and leaves hardware idle.

MegaFold attacks the problem on several fronts at once. It uses a leaner kernel for the heavy attention step, shares the pairwise data across cards more cheaply, fuses key operations, and keeps the host side from stalling the cards. On both major brands of graphics hardware, the system trained with much longer protein chains and cut total training time by well over half in their tests.

Catalog record

CREATORHoa La 1 §, Ahan Gupta §, Alex Morehead, Jianlin Cheng, Minjia Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20686

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MegaFold: System-Level Optimizations for Accelerating Protein Structure Prediction Models Hoa La 1 §, Ahan Gupta §, Alex Morehead, Jianlin Cheng, Minjia Zhang Abstract Protein structure prediction models such as AlphaFold3 (AF3) push the frontier of biomolecular modeling by incorporating science-informed architectural changes to the transformer architecture. However, these advances come at a steep system cost, introducing: compute- and memory-intensive operators, 2D attention mechanisms, and retrieval-augmented data pipelines, which collectively hinder the scalability of AF3 training. In this work, we present MegaFold, a cross-platform system to accelerate AF3 training. MegaFold tackles key bottlenecks through ahead-of-time caching to eliminate GPU idle time from the retrieval-augmented data pipeline, Triton-based kernels for memory-efficient EvoAttention on heterogeneous devices, and deep fusion for common and critical small operators in AF3. Evaluation on both NVIDIA H200 and AMD MI250 GPUs shows that MegaFold reduces peak memory usage of AF3 training by up to 1.23 × \times and improves per-iteration training time by up-to 1.73 × \times and 1.62 × \times respectively. More importantly, MegaFold enables training on 1.35 × \times longer sequence lengths compared to PyTorch baselines without running out-of-memory, significantly improving the scalability of modern protein folding models. We open source our code at https://github.com/Supercomputing-System-AI-Lab/MegaFold/ . \textsuperscript{1} \textsuperscript{1} footnotetext: Work done while interning at UIUC § § footnotetext: Equal contribution I Introduction Recent advancements in protein structure prediction have been driven by breakthroughs in deep learning models such as AlphaFold2 [ 1 ] and AlphaFold3 [ 2 ] . These models not only achieve near-experimental accuracy but also open new avenues for understanding complex biomolecular interactions [ 3 , 4 , 5 , 6 , 7 , 8 ] . In particular, AlphaFold3 (AF3) takes a major step forward by generalizing structure prediction to protein-protein, protein-DNA/RNA, protein-ligand, and protein–ion interactions by incorporating template structures and experimental data, making a shift towards science-informed model design. However, AF3 introduces significant system-level challenges. The model employs a retrieval-augmented preprocessing pipeline that incorporates evolutionary context information, novel 2D attention mechanisms, and a diffusion-based output module. These innovations, while biologically motivated, create execution bottlenecks that limit the training efficiency and scalability of AF3 on modern GPUs. Our profiling of a popular open-source AF3 implementation [ 9 ] (explained in Section III ) uncovers that AF3’s science-informed architecture leads to new system performance bottlenecks such as out-of-memory error when scaling sequence lengths and sub-optimal training speed. Moreover, a root cause analysis shows that these challenges are hard to be resolved through prior training frameworks and optimizations such as Megatron-LM [ 10 ] , DeepSpeed [ 11 ] , and FlashAttention [ 12 ] , which are designed for canonical transformer architectures used in LLMs. Concretely, the key challenges in training AF3 are as follows. First, the AF3 training pipeline incurs expensive CPU-side data preprocessing. Unlike LLMs whose data pipeline often includes solely of input reading and tokenization, AF3 training pipelines include heavy processing of searching similar sequences across different species–a process known as Multiple Sequence Alignment (MSA)–to provide rich evolutionary context [ 1 ] . The processing is conducted on the CPU, which introduces significant GPU idle time and makes it a major performance bottleneck during training (Section III ). Second, AF3 employs a unique yet memory-heavy 2D attention mechanism. Similar to how LLMs use transformers to learn relationships between word tokens in a sentence, AF models adapt transformers to capture interactions among amino acids or nucleic acids in a biological sequence such as protein, DNA, or RNA, through an evolutionary attention mechanism, which we term EvoAttention . However, this attention mechanism performs attention calculation between the pair-wise representation of amino-acids and injects priors through additive biases, thus operating on 2D data (amino-acid pairs) instead of 1D data as in natural language (sequences). The 2D EvoAttention has a O({N_{token}}^{3}) memory complexity, leading to peak memory explosion, preventing scientists from training longer sequences (Section III ). In addition, because the 2D EvoAttention is different from the standard attention mechanism in Transformers, optimizations such as FlashAttention [ 12 , 13 , 14 ] do not directly apply. Third, AF3 incurs prolonged execution from many small but frequent and memory-bound operators. Unlike LLMs consuming token embeddings as the only input, AF models must capture the intricate geometry of molecules, thereby requiring multiple interdependent input features (MSAs, single and pairwise representations, atom-level embeddings, etc.) This complexity results in a large number of small but memory-bound operators (e.g. layer-normalizations, linear layers, activation functions) to support its modeling capabilities. This adversarially impacts performance due to kernel launch overhead and low memory-bandwidth utilization, and increases memory consumption due to many intermediate activations stored for the backward pass. Collectively, these issues constrain training AF3 models to short protein lengths and cause elongated training times (e.g., 11 days on 128 GPUs [ 15 ] despite being a moderate \sim 500M parameter model). Motivated by these challenges, we develop MegaFold, a novel system that enhances the efficiency and scalability of training protein structure prediction models, such as AF3. MegaFold achieves this by a targeted set of optimizations to mitigate the compute and memory heavy operators introduced in protein structure prediction models. It consists of an ahead-of-time cache-based data-loader (Section IV-A ), memory-efficient kernels for EvoAttention (Section IV-B ), and novel fusions of small but frequent AlphaFold-centric operators (Section IV-C ). We implement MegaFold by extending a popular open-source PyTorch-based AF3 training system [ 9 ] , specifically designing MegaFold to be portable across GPU platforms. We evaluate our system on both NVIDIA H200 and AMD MI250 hardware. Our results show that MegaFold reduces the peak memory usage by up to 1.23 \times , while improving per-iteration training time by up to 1.73 \times and 1.62 \times on NVIDIA and AMD hardware respectively, over existing solutions. Moreover, our optimizations enable 1.35 \times longer input sequence lengths compared to the PyTorch baseline. These improvements make it more feasible to train advanced protein structure prediction models on diverse and modern GPU hardware. II Background In this section, we introduce key components of protein structure prediction models and contemporary techniques to train such models at scale. We contextualize our discussion to AF3 [ 2 ] , the latest protein structure prediction model. II-A AlphaFold3 Architecture AlphaFold3 [ 2 ] is a state-of-the-art structural biology model, which extends the capabilities of its predecessor, AlphaFold2 [ 1 ] , to support not just protein folding, but also interactions among protein, DNA, RNA, and small molecules. In particular, AF3 incorporates a significantly more complex architecture and data pipeline to generalize across biomolecular systems. Fig. 6 illustrates the high-level architecture of AF3. Below we introduce the necessary background needed for understanding subsequent sections (Simon and Silberg [ 16 ] contains a more in-depth explanation of the AlphaFold3 architecture for the interested reader). II-A 1 AlphaFold3 Inputs and Data Pipeline AlphaFold3 begins by processing raw biomolecular inputs, such as proteins, DNA, and small molecules, into two main internal representations: a single representation for each token and a pair representation for token interactions. These become the key inputs to subsequent modules in AF3. The pipeline starts by decomposing each input complexes into individual atoms using tools such as RDKit [ 17 ] . Local atomic features are then aggregated to create the

Read the complete text