Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O’Bryan Mathematics Competition
Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou · 2026
Read Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O’Bryan Mathematics Competition by Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31048.txt.
Introduction
Gaurab Baral, Aaditya Khanal, Yangyang Tao and Junxiu Zhou distil a large reasoning model's skill into a compact student, using a local mathematics competition as the test bed. The teacher, DeepSeek-R1, scored 91.40% on historical John O'Bryan Mathematics Competition problems from 2011 to 2025, while the base student, Qwen2.5-7B, scored 64.67%. A dual-agent framework built a chain-of-thought training corpus from the teacher, and the student was fine-tuned with low-rank adaptation on Apple Silicon. A first long run overfit badly: validation loss bottomed out at 200 iterations, then rose.
Guided by that, the team ran five training runs capped at 200 iterations with varied seeds. The student averaged 69.43% on the competition set, 4.76 points above its base, with only 0.17 points of spread between runs, and reached 73.1% on the MATH-500 benchmark. Longer permitted responses scored far better: accuracy fell from 69.43% at the richest reasoning level to 41.9% at the leanest.
Catalog record
| CREATOR | Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O’Bryan Mathematics Competition Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou Abstract This paper investigates knowledge distillation from a large reasoning model (DeepSeek-R1) to a compact student model (Qwen2.5-7B). Using historical problems from the John O’Bryan Mathematics Competition at Northern Kentucky University (2011–2025), we build a Chain-of-Thought (CoT) training corpus through a dual-agent framework. The dataset is used to fine-tune the student model with Low-Rank Adaptation (LoRA) on Apple Silicon hardware using the MLX framework [ 1 ] . The base Qwen2.5-7B model achieves 64.67% accuracy on competition problems, while the DeepSeek-R1 teacher achieves 91.40%. An initial 1,000-iteration training run revealed severe overfitting, with validation loss reaching a minimum at iteration 200 before rising steadily. Based on this finding, we ran five independent training runs each limited to 200 iterations with varied random seeds to assess result stability. Across these five runs, the fine-tuned student model achieves a mean accuracy of 69.43% ± \pm 0.17% on the competition dataset, a 4.76 percentage-point improvement over the base model, and generalizes to 73.1% ± \pm 0.18% on the MATH-500 benchmark. We further study how response length affects answer quality across six reasoning levels (R1–R6): accuracy declines consistently from 69.43% at R1 (mean 220 words) to 41.9% at R6 (mean 31.2 words), with the two-person speed section most sensitive to token reduction. These results demonstrate that CoT distillation improves compact student models and that response length is a critical factor in mathematical reasoning quality. 1 Introduction Large language models (LLMs) capable of advanced mathematical reasoning have become widely used in recent years. Models such as DeepSeek-R1 reach strong performance on competition-level mathematics benchmarks, but their size makes local deployment impractical. Knowledge distillation offers a way to address this: by transferring the reasoning behavior of a large teacher model into a smaller student model, one can produce compact models that approach the teacher’s performance at substantially lower computational cost [ 2 ] . Despite recent progress in distilling reasoning ability from large models [ 3 , 4 ] , most studies rely on generic benchmarks such as MATH-500 [ 5 ] or GSM8K [ 6 ] , which may include data seen during pretraining. There is a gap in understanding how distillation performs on domain-specific, expert-curated problem sets with lower contamination risk. This work addresses that gap by applying knowledge distillation to undergraduate mathematics competition problems using the John O’Bryan Mathematics Competition at Northern Kentucky University (NKU) as our problem corpus. The competition covers multiple difficulty levels and mathematical topics, making it a focused and well-defined dataset for evaluating LLM reasoning. To our knowledge, this is the first study to apply CoT distillation to this corpus and to quantify the effect of token budget constraints across six reasoning levels on competition-style problems. Our main contributions are as follows: (1) we construct a CoT training corpus from 15 years of competition problems using a dual-agent teacher-verifier pipeline; (2) we demonstrate that LoRA fine-tuning with early stopping improves Qwen2.5-7B from 64.67% to 69.43% \pm 0.17%, generalizing to MATH-500; (3) we quantify how accuracy degrades as token budgets are reduced across six levels, identifying a practical lower bound of roughly 50–100 words for multi-step competition problems; and (4) we provide an error-type analysis showing that roughly 40% of failures are formatting rather than mathematical errors. The pipeline proceeds as follows. First, we collect and digitize competition problems from 2011 to 2025. Second, a dual-agent framework powered by the DeepSeek-R1 API produces CoT reasoning traces and verifies answers. Third, a quantized Qwen2.5-7B student model is fine-tuned with LoRA using the MLX framework. A diagnostic 1,000-iteration run identifies the best early stopping point, after which five independent 200-iteration runs with different random seeds produce stable results. Finally, we evaluate the fine-tuned models on both the competition dataset and MATH-500, and run a multi-level token budget experiment. The full implementation, including data preparation scripts, fine-tuning code, evaluation pipelines, and raw result files, is publicly available at https://github.com/TempGaurab/Distillation.John-O-Bryan . 2 Related Work 2.1 Knowledge Distillation Knowledge distillation was introduced by Hinton, Vinyals, and Dean [ 4 ] , who showed that a smaller network can approximate a larger ensemble by training on soft output targets. Soft targets carry richer class-relationship information than one-hot labels, helping the student generalize beyond the training distribution. Gou et al. [ 2 ] survey the extensive literature that followed, documenting how distillation techniques have expanded to cover feature-level, relation-based, and task-specific objectives across many model types. Boix-Adsera [ 7 ] provides PAC-theoretic justification, proving that student models trained from a teacher can achieve lower sample complexity than learning from raw labels alone. 2.2 Distillation for Reasoning LLMs Applying distillation to reasoning-focused LLMs has gained traction since 2023. DeepSeek-R1 [ 3 ] reached state-of-the-art performance on MATH-500 and AIME benchmarks using reinforcement learning, and its technical report demonstrates that distilled smaller variants (e.g., DeepSeek-R1-Distill-Qwen-32B) retain much of the teacher’s reasoning ability. Deng et al. [ 8 ] propose implicit chain-of-thought distillation, where teacher reasoning is encoded into student hidden states rather than surface-level text, improving GSM8K performance without generating explicit intermediate steps. Ho et al. [ 9 ] show that fine-tuning smaller GPT-2 variants on teacher-generated CoT traces substantially improves multi-step arithmetic accuracy, providing early evidence that reasoning traces are effective training signal. Magister et al. [ 10 ] extend this finding to T5 models of varying sizes, showing that even 11B-parameter students benefit from distilled CoT supervision on grade-school and symbolic reasoning tasks. Parameter-efficient fine-tuning methods such as LoRA [ 11 ] and prefix tuning have been shown to be effective for adapting LLMs to specific domains with limited data [ 12 , 13 ] . 2.3 Vision-Language Distillation Knowledge distillation has also been extended to the vision-language domain. The Vision-Language-Vision (VLV) auto-encoder framework [ 14 ] introduces a cost-efficient approach to building strong image-captioning models by using a frozen text-to-image diffusion model as an information bottleneck, then fine-tuning an LLM to decode the resulting language representations into detailed captions. The method achieves performance comparable to GPT-4o and Gemini 2.0 Flash at under $1,000 training cost, demonstrating that strategic reuse of pretrained components can dramatically reduce data and compute requirements. This principle of leveraging a frozen teacher to generate high-quality training signal without retraining parallels our use of DeepSeek-R1 as a fixed API-based teacher. 2.4 Response Length and Reasoning Quality The relationship between response length and model accuracy has received growing attention. Wei et al. [ 15 ] show that chain-of-thought prompting substantially improves performance on multi-step arithmetic and commonsense problems, establishing that explicit intermediate steps are beneficial rather than redundant. Inference-time compute scaling results [ 3 ] further confirm that allowing models more tokens to reason improves downstream accuracy on hard benchmarks. Related work on structuring intermediate reasoning steps, such as the subgoal-based demonstration learning framework of Zhao et al. [ 16 ] for formal theorem proving, similarly suggests that the organization and granularity of intermediate reasoning steps materially affects downstream task accuracy, motivating closer study of how compressing those steps under a token budget affects performance. Our token budget experiment directly measures this tradeoff across six reasoning levels