Curriculum Learning-Guided Progressive Distillation in Large Language Models

Jincheng Cao, Fanzhi Zeng, Leqi Liu, Aryan Mokhtari · 2026

Read Curriculum Learning-Guided Progressive Distillation in Large Language Models by Jincheng Cao, Fanzhi Zeng, Leqi Liu, Aryan Mokhtari (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11260.txt.

Introduction

Jincheng Cao and three co-authors propose Curriculum Learning-Guided Progressive Distillation, or CLPD, for large language models. Knowledge distillation transfers large-model skills into smaller, efficient student models. But current methods overlook learning order of training data and capacity mismatch between teacher and student. This caps performance and yields the odd pattern where stronger teachers still fail to turn out better students.

CLPD handles both factors by matching data difficulty to teacher strength. It builds an explicit curriculum that orders training examples from easy to hard, while an implicit plan over supervision signals brings in teachers of growing size in stages. The framework is modular and plugs into standard algorithms with minimal overhead. On reasoning benchmarks, CLPD consistently beats standard methods, data ordering alone, and teacher order alone across multiple settings. Users get a unified method that treats both together.

Catalog record

CREATORJincheng Cao, Fanzhi Zeng, Leqi Liu, Aryan Mokhtari
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62853
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Curriculum Learning-Guided Progressive Distillation in Large Language Models Jincheng Cao, Fanzhi Zeng, Leqi Liu, Aryan Mokhtari Abstract Knowledge distillation is a key technique for transferring the capabilities of large language models (LLMs) into smaller, more efficient student models. Existing distillation approaches often overlook two critical factors: the learning order of training data and the capacity mismatch between teacher and student models. This oversight limits distillation performance, as manifested by the counter-intuitive phenomenon where stronger teachers fail to produce better students. In this work, we propose Curriculum Learning–Guided Progressive Distillation (CLPD), a unified framework that explicitly accounts for both factors by aligning data difficulty with teacher strength. CLPD constructs an explicit curriculum by organizing training examples from easy to hard, while simultaneously applying an implicit curriculum over supervision signals by progressively scheduling teachers of increasing capacity. Our framework is modular and can be integrated into standard distillation algorithms with minimal overhead. Empirical results on the reasoning benchmarks demonstrate that CLPD consistently outperforms standard distillation, data ordering alone, and teacher scheduling alone across multiple settings. These findings highlight the importance of jointly considering data ordering and teacher capacity when distilling reasoning abilities into small language models. † † footnotetext: {}^{{}^{\dagger}} The University of Texas at Austin. {}^{{}^{\ddagger}} Google Research. Corresponding author: Jincheng Cao <[email protected]>. 1 Introduction Knowledge distillation (KD) 12 , a widely used model compression paradigm in deep learning, addresses this challenge by transferring knowledge from a high-capacity teacher model to a smaller student model. As LLMs have advanced, KD has become a central strategy for building efficient small language models that preserve much of the teacher’s capability. This trend is evident in recent systems such as Llama 3.2 23 , Gemma-2 30 , and the Deepseek-R1 distilled models 5 , which explicitly rely on distillation to balance quality and efficiency. In the LLM era, KD methods are often grouped into two broad families: white-box KD and black-box KD. White-box distillation refers to settings where the student has access to the teacher’s internal probability distributions or hidden states. Standard white-box methods align the teacher and student by matching their output distributions, often using Kullback–Leibler divergence (KLD) 26 ; 8 or by aligning their intermediate representations 28 ; 17 ; 9 . Recent studies 8 ; 1 in white-box distillation highlight the importance of on-policy learning, where the student learns from its own generated responses rather than solely imitating the teacher’s outputs. These studies show that performing reverse KLD on student-generated text promotes mode-seeking behavior and reduces exposure bias compared to teacher-forced training. However, white-box access is often impractical when the teacher is a proprietary API model (e.g., GPT-5). In this scenario, only teacher-generated texts are accessible, defining the more challenging black-box distillation setting. Black-box KD transfers knowledge through the teacher’s outputs 19 and is widely adopted because many LLMs, such as GPT-4o 14 , Claude 3.5 2 , and Gemini 1.5 29 , are proprietary and expose only prediction interfaces. Despite these advances, most schemes pay limited attention to how training data and supervision are organized during distillation. In practice, poorly matched combinations of loss functions and data selection strategies can hinder student learning, especially when there is a substantial capacity gap between teacher and student models. Most KD methods treat teacher choice as fixed and training examples as i.i.d., rather than coupling teacher capability with example difficulty. Only a small number of works, such as 35 ; 22 , explicitly address aspects of this mismatch through modified supervision or curriculum-based training strategies during distillation. Curriculum Learning (CL) exposes models to progressively more complex training examples and has been shown to improve optimization and generalization 34 ; 32 . Recently, CL has been applied to LLM distillation 22 ; however, existing approaches typically focus on ordering training data while keeping the teacher fixed, leaving the interaction between data difficulty and teacher capability unexplored. One curiously persistent phenomenon is that a better teacher does not always yield a stronger student. Prior works 24 ; 15 ; 10 ; 37 hypothesized that this arises from a capability gap 38 ; 18 between teacher and student, and proposed progressive distillation, where the student is incrementally supervised by increasingly capable teachers. However, these approaches typically schedule teachers independently of the difficulty of training examples, leaving the interaction between teacher strength and data complexity largely unexplored. Contributions. In this paper, we argue that effective LLM distillation requires aligning what the student learns (data difficulty) with who teaches it (teacher capability). Inspired by human learning, where students first master fundamentals under simpler instruction before advancing to harder material, we propose Curriculum Learning–Guided Progressive Distillation (CLPD): an approach that gradually increases both training difficulty and teacher strength, reducing mismatch in the early stages while still providing high-quality guidance on the hardest examples. Concretely, CLPD constructs a difficulty-ordered curriculum using CoT-derived complexity when available or student loss otherwise, and couples it with a teacher schedule that progresses from weaker to stronger models across curriculum stages. The method can be integrated seamlessly with the standard LLM distillation. Empirically, CLPD consistently outperforms strong distillation baselines as well as curriculum-only and progressive-only variants across different reasoning datasets. We complement these results with analyses that pinpoint when stronger teachers help and when they hurt, thereby clarifying the necessity of scheduling teachers in accordance with the data difficulty. 2 Background and Preliminaries As our goal is to develop a plug-and-play framework for knowledge distillation, we first review common distillation objectives for LLMs, and then discuss two complementary factors that strongly affect distillation performance: the order in which training examples are presented and the capacity gap between teacher and student models. 2.1 Knowledge Distillation Knowledge distillation was popularized by 12 , where the student model is trained to imitate the teacher’s token-level probability distribution. Let p_{S}^{\theta} denote the student distribution and p_{T} the teacher distribution. The standard supervised knowledge distillation loss \mathcal{L}_{\mathrm{SKD}} is defined as \displaystyle\mathcal{L}_{\mathrm{SKD}}(\theta):=\mathbb{E}_{x\sim X}\Bigl[\mathcal{D}_{\mathrm{KL}}\!\left(p_{T}(\cdot\mid x)\,\|\,p_{S}^{\theta}(\cdot\mid x)\right)\Bigr]=\mathbb{E}_{x\sim X}\left[\mathbb{E}_{y\sim p_{T}(\cdot\mid x)}\left[\log\frac{p_{T}(y\mid x)}{p_{S}^{\theta}(y\mid x)}\right]\right], where the outer expectation is taken over the input distribution and the inner expectation is taken over the teacher’s output distribution. This objective provides a dense training signal by leveraging the full token-level distribution of the teacher. In contrast to this distribution-matching objective, 19 introduced sequence-level knowledge distillation (SeqKD), which trains the student to maximize the likelihood of high-probability sequences generated by the teacher: \mathcal{L}_{\mathrm{SeqKD}}(\theta):=\mathbb{E}_{x\sim X}\left[\mathbb{E}_{y\sim p_{T}(\cdot\mid x)}\left[-\log p_{S}^{\theta}(y\mid x)\right]\right]. This can be viewed as supervised fine-tuning on teacher-generated outputs; in practice, the outputs y are typically obtained via greedy or beam decoding rather than sampling. Sequence-level distillation is one of the most widely used distillation methods for large language models, as it does not require access to teacher logits and is computationally efficient. Standard single-teacher distillation with i.i.d. training data corresponds to the Vanilla baseline shown in the last row of Fig. 1 . 2.2 Learning Order and Curriculum Learning A key challenge in KD for LLMs is the training–inference mismatch: student models are typically trained on fixed datasets that do not reflect the diverse distribution of inputs encountered at inference time 8 ; 1 . To mitigate this issue, recent on-policy distillation methods incorporate student-generated outputs (SGOs) into training, allowing the student to learn from its own trajectories and

Read the complete text