HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
Geng Zhang, Shenggan Cheng, Xuanlei Zhao, Ziming Liu, Yang You ∗ · 2025
Read HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism by Geng Zhang, Shenggan Cheng, Xuanlei Zhao, Ziming Liu, Yang You ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00394.txt.
Introduction
Geng Zhang and four co-authors speed up training of very long-sequence AI models in this two thousand twenty-five paper. Transformer training is split across many processors in a pipeline. Long inputs make attention costly and leave pipeline stages idle in bubbles. Their system, HelixPipe, splits attention work itself across stages so micro-batches overlap better. A first-in-last-out schedule balances memory while hiding data transfers.
They also recompute cheap parts instead of storing them, and chop the feed-forward blocks to cut fragmentation. Gains grow as sequences lengthen, beating existing pipelines. Data centers training long-context models feel this cost daily. The paper serves machine learning systems engineers. It offers a design, not just a tweak.
Catalog record
| CREATOR | Geng Zhang, Shenggan Cheng, Xuanlei Zhao, Ziming Liu, Yang You ∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00394
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism Thanks: ∗ Corresponding author: Yang You ([email protected]) Geng Zhang, Shenggan Cheng, Xuanlei Zhao, Ziming Liu, Yang You ∗ Abstract . As transformer sequence lengths grow, existing pipeline parallelisms incur suboptimal performance due to the quadratic attention computation and the substantial memory overhead. To relieve these challenges, we propose HelixPipe, a novel pipeline parallelism for long sequence transformer training. First, HelixPipe introduces attention parallel partition, which schedules attention computations of different micro batches across different pipeline stages in parallel, reducing pipeline bubbles. Second, it employs a two-fold first-in-last-out micro batch schedule to balance memory usage and overlap communication with computation. Additionally, HelixPipe utilizes recomputation without attention and chunked MLP to mitigate fragmentation and enable longer sequences. Experiments demonstrate that HelixPipe gains increasing advantages with longer sequence lengths, and outperforms existing methods in throughput and scalability across varying pipeline sizes, model sizes, and cluster configurations. Notably, it achieves a 26% speedup over baseline methods when training a 7B model with 128k sequence length on 64 H20 GPUs. Code is available at https://github.com/code-tunnel/Megatron-LM/tree/dev . 1. Introduction Transformer models have rapidly advanced in handling longer sequence lengths, enhancing their understanding and generation capabilities in applications such as coding assistant ( Guo et al. 2024b ) , image generation ( Kong et al. 2024 ) , AI for science ( Abramson et al. 2024 ) . However, as sequence lengths grow, the quadratic complexity of attention computation and the substantial memory footprint pose significant challenges for distributed training of transformer models. Current approaches often employ multiple orthogonal parallelisms, including data parallelism ( Xing et al. 2015 ) , tensor parallelism ( Shoeybi et al. 2019 ) , sequence parallelism ( Jacobs et al. 2024 ) and pipeline parallelism ( Narayanan et al. 2021a ) , to train transformer models efficiently ( Zheng et al. 2022 ; Yuan et al. 2024 ) . While a variety of sequence parallelisms, such as Megatron sequence parallelism ( Korthikanti et al. 2023 ) , Deepspeed Ulysses ( Jacobs et al. 2024 ) , Ring attention ( Liu et al. 2023b ; Li et al. 2023 ) , Megatron context parallelism ( NVIDIA 2025a ) , USP ( Fang and Zhao 2024 ) have been proposed to split individual activations along the sequence dimension across GPUs to alleviate the computation and memory overhead, the impact of long sequence lengths on the computation efficiency and memory footprint for pipeline parallelisms remains underexplored ( Lin et al. 2025 ; Kim et al. 2023 ) . Existing pipeline parallelisms, such as 1F1B ( Narayanan et al. 2021a ; Fan et al. 2021 ; Dubey et al. 2024 ) and zero bubble pipeline parallelism ( Qi et al. 2024 ; DeepSeek-AI 2024 ) , were designed with sequence lengths below 2048. These methods partition a transformer model to a set of consecutive layers, and maps each set to a single pipeline stage. For each training iteration, the input data batch is split to multiple micro batches to execute the model along the pipeline stages. While 1F1B executes one forward pass and one backward pass at a time for each stage, zero bubble pipeline parallelism decouples the backward pass, and executes one forward pass, one backward for input activations and one backward for model parameters to reduce the pipeline bubble and improve the computation efficiency. Though existing pipeline parallelisms perform well with short sequence lengths, they achieve suboptimal performance with longer sequence lengths for two reasons. First, because of the data dependency between pipeline stages and the optimizer synchronization, pipeline parallelisms incur inevitable pipeline bubbles to downgrade its computation efficiency. Since existing methods partition a transformer model by layers, the pipeline bubble is proportional to the execution time of transformer layers. Due to the quadratic complexity of attention computation, pipeline bubble will be dominated by the attention computation as the sequence lengths increase, leaving substantial room for improvement. Second, following the 1F1B micro batch schedule, each pipeline stage needs to stash the activations for a different number of micro batches before they are used and released in the later backward pass. Such an imbalanced memory footprint is detrimental for training long sequence transformer models, as later pipeline stages can have low memory utilization whereas the earlier pipeline stages run out of memory. While zero bubble pipeline parallelism improves the computation efficiency, the decoupled backward pass leads to the same peak memory as 1F1B for all pipeline stages. To address these challenges, we present HelixPipe, a novel pipeline parallelism for efficient transformer training with long sequence lengths. Given the observation that attention computation of a transformer layer does not rely on model parameters, HelixPipe proposes the attention parallel partition . It schedules the attention computation of the same layer for different micro batches to different pipeline stages, so that the attention computations are executed in parallel across pipeline stages and can be removed from the pipeline bubbles to improve the computation efficiency. To balance the memory footprint across pipeline stages, HelixPipe proposes a two-fold first-in-last-out (FILO) micro batch schedule . The FILO schedule first executes the forward pass for all micro batches and then finishes their backward pass in the reverse order, so that it can stash the activations for the same number of micro batches at each stage before executing their backward pass to balance the peak memory usage. Moreover, as there is not data dependency between micro batches, instead of executing one micro batch at a time, HelixPipe executes two micro batches (fold) every time to hide the communication of one micro batch with the computation of the another. In addition, HelixPipe further optimizes the memory usage by a recomputation without attention strategy along with chunked MLP. As the attention computation dominates the execution time per layer, HelixPipe recomputes the intermediate activations of all operators except for the attention before the backward pass, significantly reducing the memory overhead at a marginal performance drop with the increase of sequence length. HelixPipe adopts chunked MLP to mitigate memory fragmentation introduced by the joint impact from the recomputation strategy, the two-fold FILO schedule and the long sequence lengths. The contributions of this paper are summarized as follows: • Attention parallel partition is proposed to execute the attention computation of the same layer for different micro batches in parallel across pipeline stage to remove the heavy attention computation from pipeline bubble. • Two-fold FILO micro batch schedule is proposed to balance the memory footprint across pipeline stages and overlap the communication with computation. • Recomputation without attention strategy with chunked MLP is designed to optimize the practical memory utilization to extend the sequence length. • Empirical results show that HelixPipe has higher throughput and better scalability than baseline methods across various sequence lengths, pipeline size, model scales and GPU types. In particular, it exceeds the best baseline by 26% for training a 7B model with 128k sequence length and 64 H20 GPUs. 2. Preliminaries 2.1. Model Architecture A transformer model mainly comprises L transformer layers. This section and the following of this paper take the standard model architecture of GPT-3 ( Brown et al. 2020 ) as the targeting model for analysis. The details of