AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Ahan Gupta, Zhihao Wang, Neel DaniSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA {ag82, zhihaow6, neeld2}@illinois.edu Masahiro TanakaAnyscaleSan Francisco, CA, USA [email protected] Olatunji RuwaseSnowflakeSan Mateo, CA, USA [email protected] Minjia ZhangSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA [email protected] · 2026
Read AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism by Ahan Gupta, Zhihao Wang, Neel DaniSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA {ag82, zhihaow6, neeld2}@illinois.edu Masahiro TanakaAnyscaleSan Francisco, CA, USA [email protected] Olatunji RuwaseSnowflakeSan Mateo, CA, USA [email protected] Minjia ZhangSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27089.txt.
Introduction
Gupta and five colleagues introduce AutoSP, the first automated tool for long-context language-model training. Current libraries lack easy long-context abstractions, forcing users to rewrite code to compose sequence parallelism and other optimizations, a job needing deep expertise. AutoSP compiles models and applies sequence parallelism plus long-context-aware checkpointing, lifting trainability at negligible throughput cost. On NVIDIA hardware it extends training contexts by up to nearly three times, and on AMD hardware by about two and a half times, over hand-written baselines.
Catalog record
| CREATOR | Ahan Gupta, Zhihao Wang, Neel DaniSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA {ag82, zhihaow6, neeld2}@illinois.edu Masahiro TanakaAnyscaleSan Francisco, CA, USA [email protected] Olatunji RuwaseSnowflakeSan Mateo, CA, USA [email protected] Minjia ZhangSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37126 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27089
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism Ahan Gupta, Zhihao Wang, Neel DaniSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA {ag82, zhihaow6, neeld2}@illinois.edu Masahiro TanakaAnyscaleSan Francisco, CA, USA [email protected] Olatunji RuwaseSnowflakeSan Mateo, CA, USA [email protected] Minjia ZhangSSAIL Lab, University of Illinois Urbana-ChampaignChampaign, IL, USA [email protected] Abstract Large-language-models (LLMs) demonstrate enormous utility in long-context tasks which require processing prompts that consist of tens to hundreds of thousands of tokens. However, existing LLM training libraries do not provide easy to use abstractions to optimize for long-context training, instead focusing on optimizations for models with large parameter counts through ZeRO-3/FSDP, Tensor and Pipeline parallelism. This forces users to rewrite LLM training libraries to incorporate compositions of various complex long-context optimizations, such as sequence-parallelism, to training pipelines; a process that requires in-depth expertise, reducing developer productivity. To tackle these challenges, we introduce AutoSP: the first automated solution to automatically optimize LLM training for longer-contexts. AutoSP compiles models and applies a targeted set of optimizations: automated sequence parallelism, and long-context aware activation-checkpointing, to drastically enhance LLM trainability at negligible cost to throughput. Our evaluation demonstrates AutoSP’s capability on both NVIDIA and AMD hardware, increasing training contexts by upto 2.7 × \times and 2.5 × \times respectively over competitive hand-written baseline at negligible cost to runtime performance. 1 Introduction Large Language Models (LLMs) are increasingly being trained with long-context data for scenarios such as document understanding ( 10 ; 26 ; 15 ) , multi-step reasoning ( 6 ; 9 ) , and extended multi-turn dialogue generation ( 22 ; 2 ) . These use cases often contain input sequences ranging from tens to hundreds of thousands of tokens, creating massive activation memory demands and pushing the memory and system limits of GPU clusters. To circumvent out-of-memory errors, researchers have explored Sequence Parallelism (SP), a key enabler for long-context training. State-of-the-art SP strategies such as DeepSpeed-Ulysses ( 12 ) and RingAttention ( 17 ) distribute the sequence dimension of activations across devices and allow the training engine to leverage aggregated GPU memory to train longer contexts with increasing device counts. Despite effectively enabling long-context training, existing SP are implemented in eager mode and tightly coupled to specialized training frameworks such as DeepSpeed ( 19 ) and Megatron-LM ( 20 ) . Integrating SP to new training pipelines typically requires invasive code refactoring, which makes it difficult to apply across diverse model architectures and hardware platforms. Developers must manually insert communication collectives (e.g., all2all ) between operators that require the full input sequence (such as attention), manage activation layouts across devices, and ensure correctness in both forward and backward passes. These manual efforts reduce scientists’ productivity and limit portability. To improve productivity, researchers have begun to lift several complex distributed training strategies such as ZeRO-3/FSDP ( 19 ) into SoTA deep-learning compilers, e.g. PyTorch-2.0 ( 1 ) . Examples include: SimpleFSDP ( 25 ) & DeepCompile ( 21 ) , which implement ZeRO-3/FSDP as a series of compiler passes in the PyTorch-2.0 ecosystem. However, each of these techniques focuses on how to increase model parameter counts, uncovering different ways to shard model parameters rather than explicitly optimize for long-context training. While these efforts successfully lift data and model parallelism into compiler abstractions, they do not address parallelism strategies tailored to long-context training. This raises the question: can SP also be lifted into a deep-learning compilation stack to enable automated sequence parallelism? ⬇ 1 # Register passes # 2 reg_passes ([ ’auto_sp, sp_ac’ ]) 3 # Initialize dist training # 4 dist . init ( SP_GROUP_SIZE ) 5 # Compile model. # 6 model . compile () 7 # model training # 8 for batch in data_set : 9 output = model ( batch [:, SP_GROUP ]) 10 output . backward () 11 optimizer . step ( model ) Listing 1: AutoSP provides an easy to use interface to apply long-context optimizations to training pipelines. It manages activation layouts between devices, inserts communication collectives and ensures correctness of the forward and backward training pass. In this work, we focus on lifting SP into PyTorch-2.0’s compiler ecosystem as a compiler-based, PyTorch-native implementation. However, achieving this introduces several challenges. First, PyTorch-2.0’s compilation pipeline includes multiple intermediate-representations (IRs) such as Torch-IR, Aten-IR, and Inductor-IR, to name a few. Each IR operates at a different abstraction level and encodes the input program at varying levels of granularity. This makes identifying an appropriate IR to conduct program-analysis, so as to recover the necessary information to apply semantically-preserving rewrites that transform single-GPU code into distributed sequence-parallel execution, a major challenge. A fine-grained abstraction will make program-analysis (to uncover important attributes about the input model) increasingly non-trivial whilst a coarse-grained abstraction will make semantic-rewrites (to insert communication collectives, transform buffer sizes and recompute manually indexed tensors) infeasible. Second, inferring sequence-dependent tensor shapes for resizing intermediate buffers is non-trivial to do within a compiler. (1) The lowering process inserts intermediate data-movement operators (such as transpositions), frequently changing the sequential axis of buffers. (2) SP strategies require resizing the sequential axis of only certain buffers (such as token and position id buffers), leaving other buffers (such as attention masks) untouched. Consequently, disambiguating which token’s sequential axis requires resizing is challenging. Third, lifting SP into the PyTorch-2.0 compilation stack has consequences to other optimization passes that PyTorch-2.0 natively supports, notably: activation-checkpointing (AC). AC also enables memory savings for training by discarding activations in the forward pass and rematerializing them in the backwards pass to compute gradients. However, naively rematerializing activations together with SP triggers extraneous communication in the backwards pass, adversarially impacting runtime performance. To tackle these challenges, we introduce AutoSP, the first compiler-based, PyTorch-native implementation of sequence parallelism. AutoSP introduces two key components: (1) a sequence-parallel transformation pass that automatically inserts communication collectives and reshapes activations, and (2) a sequence-aware activation checkpointing pass that exploits the compute-memory characteristics of long-context training. With just a few lines of code (as shown in 1 ), scientists can compile standard PyTorch models into distributed long-context training pipelines that scale input lengths without manual engineering. Our evaluation demonstrates that AutoSP significantly improves trainability at negligible loss to training speed on diverse hardware backends (NVIDIA and AMD GPUs), enabling training with up to 2.7 \times longer input context lengths compared to hand-written SP implementations such as DeepSpeed-Ulysses and RingAttention. 2 Background Sequence parallel training. Sequence parallelism (SP) is a key enabler for long-context training. These strategies enable scaling input sequence lengths with increasing GPU resources by sharding input tensors and activations across the sequence dimension. Communication collectives are inserted in the forward and backward pass as necessary to correctly shuffle tokens to the desired device. A popular SP strategy, and the focus of this work, is DeepSpeed-Ulysses (Ulysses) ( 12 ) . We illustrate how Ulysses operates with 3-SP groups in Fig. 1 . Figure 1: DeepSpeed-Ulysses with 3-SP groups. alltoall operators toggle the layout of activations at attention-layer boundaries. Linear-projections operate on the partial sequence length, while attention-layers operate on a subset of the heads. First, tokens are sharded across the sequence dimension, with different devices operating on different parts of the input sequence. Next, linear projections form multiple Q/K/V heads. Since linear projections operate pointwise across the sequential