Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
Wenhao Li, Daohai Yu, Gen Luo, Yuxin Zhang, Fei Chao & Rongrong Ji, Yifan Wu, Jiaxin Liu, Ziyang Gong, Zimu Liao · 2026
Read Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts by Wenhao Li, Daohai Yu, Gen Luo, Yuxin Zhang, Fei Chao & Rongrong Ji, Yifan Wu, Jiaxin Liu, Ziyang Gong, Zimu Liao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02108.txt.
Introduction
Wenhao Li and colleagues present OOMB, a memory-efficient system. Long-context training is choked by GPU memory load, with activations as culprits since footprints grow with length. OOMB uses a chunk-recurrent frame with on-the-fly recomputation, holding memory at O(1) and shifting load to the KV cache. OOMB adds a paged manager. Async CPU offloading hides lag, and sparse attention cuts compute. Each extra 10K tokens adds a mere 10MB. Lets Qwen two point five 7B train with a 4M-token context on one GPU.
Catalog record
| CREATOR | Wenhao Li, Daohai Yu, Gen Luo, Yuxin Zhang, Fei Chao & Rongrong Ji, Yifan Wu, Jiaxin Liu, Ziyang Gong, Zimu Liao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37529 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.02108
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts Wenhao Li, Daohai Yu, Gen Luo, Yuxin Zhang, Fei Chao & Rongrong Ji, Yifan Wu, Jiaxin Liu, Ziyang Gong, Zimu Liao Abstract Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time. The primary culprits are the activations, whose memory footprints scale linearly with sequence length. We introduce OOMB, a highly memory-efficient training system that directly confronts this barrier. Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation memory footprint ( 𝒪 ( 1 ) \mathcal{O}(1) ) and shifts the primary bottleneck to the growing KV cache. To manage the KV cache, OOMB integrates a suite of synergistic optimizations: a paged memory manager for both the KV cache and its gradients to eliminate fragmentation, asynchronous CPU offloading to hide data transfer latency, and page-level sparse attention to reduce both computational complexity and communication overhead. The synergy of these techniques yields exceptional efficiency. Our empirical results show that for every additional 10K tokens of context, the end-to-end training memory overhead increases by a mere 10MB for Qwen2.5-7B. This allows training Qwen2.5-7B with a 4M-token context on a single H200 GPU, a feat that would otherwise require a large cluster using context parallelism. This work represents a substantial advance in resource efficiency for long-context LLM training. The source code is available at https://github.com/wenhaoli-xmu/OOMB . 1 Introduction Figure 1: An overview of the OOMB training framework. (Left) OOMB processes sequences chunk-by-chunk with activation recomputation, maintaining a constant activation memory footprint ( \mathcal{O}(1) ) and shifting the bottleneck to the KV cache. (Center) The growing KV cache is managed through a combination of paged memory, asynchronous CPU offloading, and page-level sparse attention. (Right) Performance benchmarks show that the integrated system drastically reduces peak GPU memory, enabling efficient training on long contexts with minimal memory overhead. Long-context modeling remains a formidable challenge in training Large Language Models (LLMs). The principal obstacle is not only training time but also the prohibitive GPU memory overhead associated with long sequences ( 22 ; 30 ) . Research indicates that significant long-context capabilities can be achieved with a surprisingly small number of training steps ( 24 ; 9 ) , and that fine-tuning on as little as 5% of the original pretraining data may suffice for commercial-grade performance ( 11 ; 4 ) . Despite this modest data requirement, the number of tokens processed in a single iteration grows dramatically. This causes GPU memory consumption from activations and the KV cache to scale linearly with context length, rapidly exhausting available resources. For instance, a model with a 4 \times GQA ( 2 ) ratio requires 64GB for the KV cache alone when processing a 256K context. This demand overwhelms an A100 GPU before even accounting for other network activations, often making 32K tokens a practical limit for single-GPU training ( 12 ; 32 ) . This immense memory pressure renders popular techniques like ZeRO3 ( 25 ) and tensor parallelism ( 26 ) insufficient. While various training-free methods for context extension exist ( 15 ; 13 ) , their practical utility is limited. Prior work has established that common evaluation metrics like perplexity and targeted retrieval are unreliable indicators of true long-range reasoning ( 10 ; 4 ) . Achieving robust performance necessitates dedicated contiguous pretraining. To this end, this paper directly confronts the central problem: how to train longer-context models with fewer resources without compromising performance. We introduce O ut O f the M emory B arrier (OOMB), a chunk-recurrent training framework that processes sequences in segments. During the forward pass, each chunk’s activations are computed and then immediately discarded. For the backward pass, they are recomputed on-the-fly. This strategy maintains a constant activation memory footprint, regardless of the total sequence length. This approach, however, shifts the primary bottleneck to the KV cache, which must be retained throughout the training step and scales linearly with context length. To overcome this challenge, OOMB integrates a system of co-designed optimizations for efficient KV cache management: • Paged KV Cache and Gradient Management. We introduce a paged memory manager, inspired by ( 16 ) , for both the KV cache and its gradients. This design eliminates expensive memory operations and mitigates fragmentation when appending new key-value pairs. • Specialized Kernels. We implement custom operators that execute all operations on the KV cache and its gradients directly, making them opaque to PyTorch’s autograd system. This avoids storing the KV cache as an activation and allows for direct gradient accumulation. • Asynchronous KV Cache CPU Offloading. We designed an asynchronous mechanism to preemptively offload the KV cache and its gradients, whose memory grows with context length, to CPU memory. The resulting data transfer latency is effectively masked by computation. • Page-level Sparse Attention. Our paged memory architecture natively supports page-level sparse attention ( 34 ; 23 ) . This reduces the computational complexity of attention and minimizes the communication overhead of our offloading mechanism. The synergy of these techniques yields exceptional GPU memory efficiency. Our results show that for every additional 10K context tokens, the end-to-end training memory overhead increases by a mere 10MB for Qwen2.5-7B. This allows training Qwen2.5-7B with a 4-million-token context on a single H200 GPU. In contrast, context parallelism methods ( 21 ; 19 ; 26 ) require large clusters for similar tasks ( 31 ) , highlighting a substantial advance in resource efficiency. Table 1: Comparison of Long-Context Training Methods. OOMB achieves \mathcal{O}(1) activation memory complexity, where N denotes the context length. This key advantage allows OOMB, when combined with other techniques, to practically enable single-GPU training on contexts exceeding 4M tokens. Method GPUs Activation Exact Attn Max Tokens SeCO ( 18 ) \mathcal{O}(1) ✓ 128K LongLoRA ( 7 ) 1 \times H200 \mathcal{O}(N) ✗ 128K ZeRO3 Offload \mathcal{O}(N) ✓ 128K Ring Flash Attention \mathcal{O}(N/8) ✓ 256K Tensor Parallelism 8 \times H200 \mathcal{O}(N/8) ✓ 256K OOMB + Dense Attn \mathcal{O}(1) ✓ 4M OOMB + Sparse Attn 1 \times H200 \mathcal{O}(1) ✗ 4M+ 2 Related Work Efficient Long-Context Extension. A core challenge in extending language models to long contexts is their failure to generalize to positional encodings not seen during pretraining ( 13 ) . While continued training can resolve this issue, the associated GPU memory costs are often prohibitive. This has motivated training-free context extension methods, which typically modify positional encodings to accommodate longer sequences ( 6 ; 13 ; 15 ; 3 ) . However, these approaches introduce significant trade-offs. Strategies based on interpolation or extrapolation of positional encodings can degrade performance on short-text tasks by reducing the resolution of positional information ( 10 ) . Others that reuse position indices achieve high scores on perplexity benchmarks but have been shown to fail on tasks requiring deep contextual understanding, indicating a superficial grasp of the extended context ( 10 ) . Given these limitations, state-of-the-art models still rely on fine-tuning to expand their context windows ( 12 ; 32 ; 20 ) . While recent advances in parameter-efficient fine-tuning (PEFT) have successfully minimized the memory footprint of model weights and optimizer states ( 36