Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective

Tokio Kajitsuka, Ukyo Honda, Sho Takase · 2026

Read Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective by Tokio Kajitsuka, Ukyo Honda, Sho Takase (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08880.txt.

Introduction

Kajitsuka, Honda, and Takase revisit the size gap in chain of thought teaching from a hands on view. Teaching moves thinking from a big teacher to a small student, but earlier work reports failure when the skill mismatch is large. The authors find teaching often harms the student below its pre teaching start, and that some prior settings, while good for proving the effect, miss real use. Under more real settings the gap does not always rule, and stronger teachers tend to win when options differ a lot in scores.

Catalog record

CREATORTokio Kajitsuka, Ukyo Honda, Sho Takase
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54608
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08880

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective Tokio Kajitsuka, Ukyo Honda, Sho Takase Abstract Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capacity gap: distillation may fail when the teacher–student capability mismatch is large. We revisit the capacity gap from a practical perspective by re-examining commonly used experimental settings. Notably, we find that CoT distillation often degrades performance compared to the student’s pre-distillation baseline, an issue obscured when only post-distillation comparisons are reported. We therefore propose a more realistic evaluation protocol and find that the impact of capacity gap effects does not consistently dominate across tasks and settings, especially when candidate teachers differ substantially in performance. Our results offer practical guidance for selecting teacher–student pairs in CoT distillation. 1 Introduction Large language models (LLMs) have demonstrated remarkable reasoning capabilities when prompted to generate intermediate reasoning steps ( 33 ; 18 , chain-of-thought (CoT) prompting;) , but generating lengthy intermediate outputs increases computational costs, posing a challenge for deployment at scale. CoT distillation addresses this by training a smaller, more efficient student model to mimic the reasoning behavior of a larger teacher ( 12 ; 19 ; 7 ; 14 ; 23 ; 28 ) . CoT distillation typically targets task-specific reasoning transfer, as opposed to general-purpose distillation during pretraining where synthetic data from stronger models is used for broad capability acquisition. A key concern is the so-called capacity gap : distillation may fail when the teacher–student capability mismatch is large ( 20 ; 3 ) , suggesting that the strongest available teacher may not always be the best choice and practitioners should instead seek an appropriately matched teacher. If true, this has significant practical implications, as it introduces a costly hyperparameter search over teacher models. In this paper, we revisit the capacity gap from a practical perspective. Our aim is not to dispute the existence of the capacity gap, but to assess how much it matters in practical deployment scenarios. We argue that commonly used experimental protocols do not adequately reflect realistic deployment scenarios, potentially leading to misleading conclusions about teacher–student selection. Specifically, we identify two critical issues: (1) prior studies compare configurations only after distillation, without verifying improvement over the pre-distillation baseline, and we find that distillation often degrades performance; (2) practices such as cross-teacher data filtering and including larger-student settings do not reflect practical use cases, potentially obscuring the true impact of teacher strength. To address these issues, we propose a revised evaluation protocol that includes pre-distillation baselines and removes these artificial constraints. Under this protocol, we find that the capacity gap does not consistently manifest across tasks and settings. Furthermore, when there is a substantial performance gap between candidate teachers, the benefits of stronger teachers outweigh the capacity gap effects. Our contributions are as follows: • We identify pitfalls in existing evaluation protocols from a practical perspective, most notably the failure to compare against pre-distillation baselines, which obscures cases where distillation degrades performance. • Through systematic experiments across multiple model families under a more realistic evaluation protocol, we demonstrate that the impact of capacity gap effects does not consistently dominate across tasks and settings; in particular, when a substantial performance gap exists between candidate teachers, the benefits of stronger teachers outweigh capacity gap effects. • We provide actionable guidelines for practitioners: verify that distillation improves over the baseline, and when it does and candidate teachers differ substantially in performance, prefer the higher-performing teacher. 2 Preliminaries 2.1 CoT Distillation CoT distillation aims to transfer the reasoning capabilities of a large teacher model to a smaller, more efficient student model. The general procedure consists of two stages. Rationale Generation. Given an original training dataset \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N} with ground-truth labels y_{i} , the teacher model \mathcal{T} generates reasoning chains (rationales) \{\hat{r}_{i}\}_{i=1}^{N} along with predicted answers \{\hat{y}_{i}\}_{i=1}^{N} . To ensure the quality of reasoning chains, only samples where the predicted answer matches the ground-truth label are retained: \mathcal{D}_{\text{CoT}}=\{(x_{i},\hat{r}_{i},\hat{y}_{i})\mid(x_{i},y_{i})\in\mathcal{D},\ \hat{y}_{i}=y_{i}\}. (1) Student Fine-tuning. The student model \mathcal{S} is fine-tuned on the distillation dataset \mathcal{D}_{\text{CoT}} to learn to produce similar reasoning chains. Formally, the student is trained to minimize the negative log-likelihood: \mathcal{L}=-\sum_{(x,\hat{r},\hat{y})\in\mathcal{D}_{\text{CoT}}}\log P_{\mathcal{S}}(\hat{r},\hat{y}\mid x), (2) where P_{\mathcal{S}} denotes the probability distribution of the student model. This hard distillation can be viewed as a special case of sequence-level knowledge distillation ( 17 ) , where the teacher distribution is approximated via sampling rather than logit-based KL divergence ( 11 ) . Hard distillation is the dominant paradigm in CoT distillation because teacher logits are often unavailable from proprietary models, and tokenizer mismatches across model families preclude logit-level transfer ( 12 ; 19 ; 14 ; 23 ; 28 ) . Consequently, capacity gap analyses developed for logit-based KD ( 22 ; 16 ) do not straightforwardly apply, motivating independent empirical investigation under hard distillation. 2.2 Capacity Gap The capacity gap refers to the empirical observation that distillation may yield inappropriate results when there is a substantial capability mismatch between the teacher and student models. Such mismatch can arise from various sources, including differences in model size or in reasoning style (e.g., short vs. long chain-of-thought); 20 examine both settings. This phenomenon has been reported in conventional knowledge distillation ( 24 ; 4 ) and has also been observed in CoT distillation ( 20 ; 3 ) , suggesting that using CoT from the strongest available teacher may not always be optimal. However, as an empirical phenomenon, the capacity gap is inherently dependent on experimental settings, and its practical significance remains unclear. In the following sections, we focus specifically on the hard CoT distillation setting described above and examine whether commonly used evaluation protocols adequately reflect practical scenarios, assessing how much the capacity gap actually matters in realistic deployment conditions. 3 Pitfalls in Existing Evaluation Protocols We critically examine experimental protocols commonly used in prior capacity gap studies ( 20 ; 3 ) and identify three pitfalls that may lead to conclusions that do not directly translate to practical deployment scenarios. (a) Results under the small–large setting. (b) Results under the short–long setting. Figure 1: Performance comparison on five math benchmarks (AIME, AMC, GSM8K, MATH, and OlympiadBench). We compare the pre-distillation baseline with distilled students. (a) Small–large setting: students distilled from Qwen2.5-3B-Instruct (Small Teacher) and Qwen2.5-72B-Instruct (Large Teacher). (b) Short–long setting: students distilled from Qwen2.5-32B-Instruct (Short Teacher) and QwQ-32B-Preview (Long Teacher). 3.1 Pitfall 1: Lack of Pre-Distillation Baseline Comparison Prior studies on the capacity gap ( 20 ; 3 ) compare different distillation configurations only after distillation, without examining whether distillation itself improves upon the student model’s pre-distillation performance. This omission is problematic because recent LLMs already possess substantial reasoning capabilities and may already perform well on target tasks without fine-tuning. Therefore, verifying that distillation actually improves performance is essential for its practical adoption. Experimental Setup. To examine whether prior experiments constitute a valid scenario where distillation is expected to be beneficial, we add comparison with pre-distillation baselines to the experimental settings of 20 . Our experiments are based on their publicly available code and data to ensure consistent experimental settings. 1 1 1 https://github.com/Small-Model-Gap/Small-Model-Learnability-Gap Below, we briefly summarize the experimental setup; see Appendix A for more details, including models

Read the complete text