From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models

Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro · 2025

Read From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models by Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06214.txt.

Introduction

Chejian Xu and co-authors give a recipe for training language models that can read extremely long inputs, up to four million tokens, in a 2025 paper. Long reading helps with books, videos, and tasks where the model must reason over a whole archive at once. Most models start with far smaller windows, so the skill must be added without breaking what they already do well.

Their recipe extends a tuned open model step by step, then polishes it with instruction training to keep its following and reasoning sharp. The resulting eight-billion-parameter model led long-context benchmarks while staying competitive on standard ones. The authors detail which scaling choices and data mixes mattered most, and released the model weights.

Catalog record

CREATORChejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06214

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro Abstract Long-context capabilities are essential for a wide range of applications, including document and video understanding, in-context learning, and inference-time scaling, all of which require models to process and reason over long sequences of text and multimodal data. In this work, we introduce a efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens. Our approach leverages efficient continued pretraining strategies to extend the context window and employs effective instruction tuning to maintain the instruction-following and reasoning abilities. Our UltraLong-8B , built on Llama-3.1-Instruct with our recipe, achieves state-of-the-art performance across a diverse set of long-context benchmarks. Importantly, models trained with our approach maintain competitive performance on standard benchmarks, demonstrating balanced improvements for both long and short context tasks. We further provide an in-depth analysis of key design choices, highlighting the impacts of scaling strategies and data composition. Our findings establish a robust framework for efficiently scaling context lengths while preserving general model capabilities. We release all model weights at https://ultralong.github.io/ . 1 Introduction Large language models (LLMs) have demonstrated remarkable performance across a diverse range of text and multimodal tasks ( Hurst et al., 2024 ; Gemini et al., 2024 ; Liu et al., 2024a ; Yang et al., 2024 ; Dai et al., 2024 ) . However, many applications, such as document and video understanding ( Gemini et al., 2024 ; Azzolini et al., 2025 ) , in-context learning, and inference-time scaling ( Guo et al., 2025 ) , demand the ability to process and reason over extremely long sequences of tokens ( Gemini et al., 2024 ; Xu et al., 2023 ; Yang et al., 2025 ; Xu et al., 2024b ; Lu et al., 2024 ) . In these scenarios, the limited context window of LLMs poses a significant bottleneck, as critical information scattered across lengthy documents may be overlooked. This limitation motivates the need for models that can efficiently handle ultra-long contexts without sacrificing performance on standard tasks. Recent trends in both industry and academia have focused on extending the context windows of LLMs. Proprietary systems like GPT-4o ( Hurst et al., 2024 ) support context lengths of up to 128K tokens, while the reasoning model o1 ( Jaech et al., 2024 ) further pushes this limit to 200K tokens to accommodate inference-time scaling. Other models, such as Claude 3.5 Sonnet ( Anthropic, 2024 ) and Gemini 1.5 Pro ( Gemini et al., 2024 ) , have demonstrated the feasibility of handling even larger contexts. Despite significant progress in open-access models, many efforts have been limited by the lack of detailed training data blends ( Dubey et al., 2024 ) and efficient extension recipes ( Gao et al., 2024 ) . Moreover, evaluations of these models have often relied on synthetic benchmarks that do not fully capture their performance on real-world long-context tasks ( Pekelis et al., 2024 ) . Figure 1: Overview of our training pipeline. In Stage 1, the model’s context window is extended through continued pretraining, leveraging techniques such as special document separators and YaRN-based scaling to handle ultra-long sequences. In Stage 2, instruction tuning is applied using a curated dataset to enhance the model’s instruction-following and reasoning capabilities. This pipeline enables the development of language models that achieve good performance on both long-context and standard benchmarks. In this work, we present a systematic recipe for training ultra-long context language models. Our approach involves two key stages. The first stage, continued pretraining, extends the context window of LLMs to ultra-long lengths (up to 1M, 2M, and 4M tokens) by leveraging a specially curated corpus. We introduce techniques such as the use of special document separators during concatenation and apply YaRN-based ( Peng et al., 2023b ) RoPE scaling to improve the model’s ability to process long sequences. The second stage, instruction tuning, refines the model’s instruction-following and reasoning capabilities using a high-quality, short-context supervised fine-tuning (SFT) dataset across general, mathematical, and coding domains. We validate our approach using Llama-3.1-8B-Instruct with 128K context window ( Dubey et al., 2024 ) as the starting point and conduct extensive evaluations on both synthetic and real-world long-context benchmarks, including RULER ( Hsieh et al., 2024 ) , LV-Eval ( Yuan et al., 2024b ) , and InfiniteBench ( Zhang et al., 2024 ) . Additionally, we benchmark our models on standard datasets such as MMLU ( Hendrycks et al., 2020 ) , MATH ( Hendrycks et al., 2021 ) , GSM-8K ( Cobbe et al., 2021 ) , and HumanEval ( Chen et al., 2021 ) to ensure that the extended context does not compromise general task performance. Evaluation results demonstrate that our final models, UltraLong-8B , achieve state-of-the-art (SOTA) long-context performance while maintaining competitive performance on standard benchmarks. Our contributions are summarized as follows: • We propose an efficient and scalable training recipe that extends the context window of LLMs to ultra-long lengths (up to 4M tokens) while preserving, and in some cases enhancing, performance on standard benchmarks. • We introduce techniques such as the use of special document separators during data preparation and apply YaRN-based scaling for positional embeddings, both of which are shown through ablation studies to be essential for effective long-context modeling. • We show that a one-step continued pretraining strategy is more efficient than a multi-step approach for context extension, consistently yielding superior results on both synthetic and real-world long-context benchmarks. • We conduct extensive experiments on benchmarks including RULER, LV-Eval, InfiniteBench, MMLU, MMLU-Pro, MATH, GSM-8K, and HumanEval, showing that our UltraLong-8B models outperform existing baselines in both long-context and standard tasks. The remainder of the paper is organized as follows. Section 2 reviews related work on long-context language modeling and extension strategies. Section 3 details our training methodology, including both the continued pretraining and instruction tuning stages. In Section 4 , we describe the baseline models and evaluation benchmarks used in our experiments, while Section 5 presents our experimental results. Section 6 provides an in-depth analysis through ablation studies, and Sections 7 and 8 concludes the paper by discussing limitations and outlining directions for future research. 2 Related Work 2.1 Long-context methods Existing context extension strategies for long-context language models can be broadly categorized into three groups: exact attention methods, approximate attention methods, and approaches that incorporate additional modules. Exact attention methods enhance the parameterization of the attention mechanism to support longer sequences. Techniques such as Position Interpolation (PI) ( Chen et al., 2023b ) , NTK-aware ( bloc97, 2023 ) , Dynamic NTK ( emozilla, 2023 ) , YaRN ( Peng et al., 2023b ) , and CLEX ( Chen et al., 2023a ) —all based on RoPE ( Su et al., 2024 ) —design position embeddings that enable length extension. These approaches can be applied either through fine-tuning or to frozen models. In contrast, approximate attention methods adopt structured approximations to mitigate the computational cost of long-context processing. For example, LongLoRA ( Chen et al., 2023c ) combines LoRA ( Hu et al., 2021

Read the complete text