Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling

Hongtao Xu, Wenting Shen, Yuanxin Wei, Ang Wang, Guo Runfan, Tianxing Wang, Yong Li, Mingzhen Li, Weile Jia · 2025

Read Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling by Hongtao Xu, Wenting Shen, Yuanxin Wei, Ang Wang, Guo Runfan, Tianxing Wang, Yong Li, Mingzhen Li, Weile Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19609.txt.

Introduction

Hongtao Xu and co-authors present Skrull in a 2025 paper, a scheduler that makes long context fine tuning of language models faster. Teaching a model to handle long texts means training on a mix of long and short samples. That mix is awkward for training systems, which cannot be efficient for both lengths at once. Total training time suffers as a result.

Skrull schedules data on the fly, balancing the compute load of long and short samples as training proceeds. The team frame scheduling as a joint optimization problem and study its trade offs. Their scheduling method is light enough to run online at near zero cost. Built on a leading distributed training system, Skrull beat that system by several times on average in real long context jobs. The largest gains in their tests were higher still.

Catalog record

CREATORHongtao Xu, Wenting Shen, Yuanxin Wei, Ang Wang, Guo Runfan, Tianxing Wang, Yong Li, Mingzhen Li, Weile Jia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling Hongtao Xu, Wenting Shen, Yuanxin Wei, Ang Wang, Guo Runfan, Tianxing Wang, Yong Li, Mingzhen Li, Weile Jia Abstract Long-context supervised fine-tuning (Long-SFT) plays a vital role in enhancing the performance of large language models (LLMs) on long-context tasks. To smoothly adapt LLMs to long-context scenarios, this process typically entails training on mixed datasets containing both long and short sequences. However, this heterogeneous sequence length distribution poses significant challenges for existing training systems, as they fail to simultaneously achieve high training efficiency for both long and short sequences, resulting in sub-optimal end-to-end system performance in Long-SFT. In this paper, we present a novel perspective on data scheduling to address the challenges posed by the heterogeneous data distributions in Long-SFT. We propose Skrull, a dynamic data scheduler specifically designed for efficient long-SFT. Through dynamic data scheduling, Skrull balances the computation requirements of long and short sequences, improving overall training efficiency. Furthermore, we formulate the scheduling process as a joint optimization problem and thoroughly analyze the trade-offs involved. Based on those analysis, Skrull employs a lightweight scheduling algorithm to achieve near-zero cost online scheduling in Long-SFT. Finally, we implement Skrull upon DeepSpeed, a state-of-the-art distributed training system for LLMs. Experimental results demonstrate that Skrull outperforms DeepSpeed by 3.76x on average (up to 7.54x) in real-world long-SFT scenarios. 1 Introduction Long-context capabilities are important for large language models (LLMs) to handle various tasks such as long document summarization, question answering, multi-turn dialogue and code generation. Mainstream LLMs such as Llama [ 21 , 10 ] , Qwen [ 19 ] and GPT-4 [ 18 ] can support the context window of up to 128K tokens. Google’s Gemini [ 9 ] can even achieve up to 1M tokens per context window. Typically, additional training phases like long-context supervised fine-tuning (Long-SFT) as well as long-context continue pre-training (Long-CPT) are employed to extend the context length. For example, Llama3 [ 10 ] is fine-tuned with 99.89% short sequence (averaging under 1K tokens) and 0.11% long sequence (averaging around 37K tokens). Qwen2.5-Turbo [ 19 ] gradually extends context length by training on 40% long sequences and 60% short sequences. Training on those meticulous gathered datasets enables smoothly adaptation of LLMs to longer context while still maintaining the performance on short context tasks. However, this heterogeneous data distribution in Long-SFT poses significant challenges for existing distributed LLM training frameworks [ 14 , 20 , 15 ] , exhibiting sub-optimal efficiency. For instance, the heterogeneous data distribution poses a dilemma for parallelism and memory-reduction strategies. Specifically, long sequences necessitate context parallelism and other memory-reduction approaches due to their tremendous memory requirements. However, those approaches compromise the training efficiency for short ones due to the overheads like unnecessary communication and GPU under-utilization. Moreover, the wide sequence length distribution in long-SFT worsen the mismatch of computation characteristics in Attention module, which exhibit quadratic computational complexity and linear memory consumption [ 7 , 6 ] , leading to another dilemma for load balance problem. To tackle the above challenges, we propose Skrull, a dynamic data scheduler dedicated for Long-SFT scenarios. Skrull efficiently handle the unique data distributions in Long-SFT scenario through two main components: Distributed-Aware Context Parallelism (DACP) and Global Data Scheduling (GDS). DACP selectively shards sequences and schedules them across different workers to minimize the performance degradation while maintains the ability of handling long sequence. GDS enlarge the scope of scheduling and improve the GPU utilization during training. The two components collaborate with each other at different scheduling granularities. Furthermore, to achieve the optimal performance, we formulate the scheduling process as a joint optimization problem and design a lightweight heuristic algorithm to solve it at runtime. Experimental results demonstrate that Skrull improves the end-to-end training performance by 3.76x on average (up to 7.54x) compared to DeepSpeed, a state-of-the-art distributed LLM training framework. Our key contributions are summarized as follows: • We provide a new perspective of data scheduling to address the heterogeneous sequence length distribution. • We propose a new context parallelism called DACP based on fine-grained data scheduling, which maintaining both the processing capabilities for long sequences and efficiency for short sequences, enabling efficient training on heterogeneous data distribution in long-SFT scenario. • We implement coarse-grained global data scheduling (GDS) and further formulate GDS and DACP as a joint optimization problem through performance modeling. • We design a lightweight heuristic algorithm and achieve performance gains by 3.76x on average (with a peak improvement of 7.54×) in real-world datasets. 2 Preliminaries Data Parallelism (DP). Data parallelism [ 15 , 25 , 20 ] partitions the training samples to multiple workers and each worker maintains a complete model weight replica. In each iteration, all workers process a subset of global batch independently and then synchronize the gradients across all DP ranks. However, due to the inherent synchronization semantic in DP, the load balance becomes a noticeable problem, especially in long context scenarios. Context Parallelism (CP). Context parallelism partitions the input tensor along the sequence length dimension and distributes it to multiple workers [ 12 , 16 , 10 ] . CP is emerging as an inevitable parallel strategy when handling long context. In the Transformer architecture, the primary challenge of CP stems from the parallelization of Attention module because each tokens needs to attend to other tokens in the sequence. Consequently, the communication in CP is inevitable. Notably, DACP, proposed in this paper, leverages data scheduling to minimize the overheads caused by CP and is orthogonal to specific CP implementations. 3 Observation 3.1 Heterogeneous Sequence Length Distribution As shown in Figure 1(a) , we observe pronounced variance in the sequence length distribution across real-world Long-SFT datasets, including Wikipedia [ 2 ] , LMsysChat1M [ 26 ] and ChatQA2-Long-SFT [ 1 ] . Among them, the sequence length distribution of ChatQA2-long-SFT exhibits a bimodal pattern, where the proportions of long and short sequences are nearly equal. Specifically, approximately 40% of sequences are shorter than 8K tokens, while the remaining 60% exceed this threshold. As comparison, long-tail distributions represent another typical pattern in Long-SFT datasets. In Llama3’s internally collected Long-SFT datasets [ 10 ] , we find that 99.89% of sequences are under 1K tokens on average, while the remaining 0.11% are approximately 37K tokens, showcasing extremely skewed long-tail distribution. Due to data accessibility constraints, we plot the sequence length distribution of Wikipedia and LMsysChat1M in Figure 1(a) , which have the identical feature with Llama3’s Long-SFT dataset. Table 1 lists the portions under different lengths thresholds for these three datasets, highlighting the heterogeneous sequence length distribution in Long-SFT. (a) Sequence length distribution. (b) Performance (FLOPS) with different CP workers. Figure 1 : Sequence length distribution on different datasets, and corresponding performance impact. Table 1 : Percentage of sequence length in real-world datasets. Dataset <1K <4K <8k <32K <128K Longest Wikipedia 87.88% 99.34% 99.92% 99.99% 100.0% 78K LMsysChat1M 87.12% 99.35% 99.87% 99.98% 99.99% 1643K ChatQA2-Long-SFT 21.92% 31.48% 40.43% 99.86% 100.0% 99K 3.2 Performance Degradations for Short Sequences In this section, we discuss our observation on the performance degradations and GPU under-utilization for short sequences in Long-SFT training. During the training process, the context parallelism degree and other memory reduction strategies such as gradient accumulation

Read the complete text