Small Models Struggle to Learn from Strong Reasoners
Yuetai Li Xiang Yue Zhangchen Xu Fengqing Jiang Luyao Niu, Bill Yuchen Lin Bhaskar Ramasubramanian Radha Poovendran · 2025
Read Small Models Struggle to Learn from Strong Reasoners by Yuetai Li Xiang Yue Zhangchen Xu Fengqing Jiang Luyao Niu, Bill Yuchen Lin Bhaskar Ramasubramanian Radha Poovendran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12143.txt.
Introduction
Yuetai Li and co-authors find a learnability gap for small language models in a 2025 paper. Large models reason well, and teaching small models from their long chains of thought seemed promising. Yet small models with three billion parameters or fewer do not gain steadily from those long chains. Shorter, simpler reasoning fits their capacity better, the team shows.
Their fix, Mix Distillation, blends long and short chains, or chains from large and small models, in one training mix. That balance lifts small model reasoning well above training on either kind alone, they report. Direct distillation from strong reasoners has clear limits, they argue.
Catalog record
| CREATOR | Yuetai Li Xiang Yue Zhangchen Xu Fengqing Jiang Luyao Niu, Bill Yuchen Lin Bhaskar Ramasubramanian Radha Poovendran |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12143
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Small Models Struggle to Learn from Strong Reasoners Yuetai Li Xiang Yue Zhangchen Xu Fengqing Jiang Luyao Niu, Bill Yuchen Lin Bhaskar Ramasubramanian Radha Poovendran Abstract Large language models (LLMs) excel in complex reasoning tasks, and distilling their reasoning capabilities into smaller models has shown promise. However, we uncover an interesting phenomenon, which we term the Small Model Learnability Gap : small models ( ≤ \leq 3B parameters) do not consistently benefit from long chain-of-thought (CoT) reasoning or distillation from larger models. Instead, they perform better when fine-tuned on shorter, simpler reasoning chains that better align with their intrinsic learning capacity. To address this, we propose Mix Distillation, a simple yet effective strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models. Our experiments demonstrate that Mix Distillation significantly improves small model reasoning performance compared to training on either data alone. These findings highlight the limitations of direct strong model distillation and underscore the importance of adapting reasoning complexity for effective reasoning capability transfer. 1 Introduction Large language models (LLMs) ( Anthropic, 2023 ; Brown et al., 2020 ; OpenAI, 2023 ; Touvron et al., 2023a ) have demonstrated remarkable performance in complex reasoning tasks, enabling advancements in mathematical problem-solving, logical inference, and structured decision-making ( Cobbe et al., 2021 ; Shao et al., 2024 ; Yang et al., 2024 ) . A key advancement in improving LLM complex reasoning capability is the chain-of-thought (CoT) prompting. This technique decomposes complex problems into intermediate reasoning steps, enhancing both performance and interpretability. ( Wei et al., 2023 ) . Figure 1: Small student models ( \leq 3B parameters) do not consistently benefit from long CoT reasoning or distillation from large teacher models. Instead, they perform better when fine-tuned on shorter CoT reasoning or distilled from smaller teachers, which better matches their intrinsic learning capacity. We term this phenomenon the Small Model Learnability Gap . However, the high computational cost of LLMs hinders their deployment on resource-constrained devices, motivating the development of smaller models that offer similar capabilities at reduced cost. A widely adopted strategy to achieve this is distillation ( Agarwal et al., 2024 ; Hinton et al., 2015b ; Kim et al., 2024a ) , where CoT sequences generated by a strong teacher model are used to fine-tune a weaker student model. Naturally, one might expect that distilling CoT sequences from stronger models would consistently improve small models’ complex reasoning capabilities ( Agarwal et al., 2024 ; DeepSeek-AI et al., 2024 ; Min et al., 2024 ; Tunstall et al., 2023 ) . However, we reveal an interesting phenomenon, which we term the Small Model Learnability Gap (Fig. 1 ): small models do not consistently benefit from the complex reasoning sequences provided by strong teachers, such as long CoT reasoning or distillation from large models. In our experiments, we observe that when small models are exposed to long and intricate reasoning traces, they struggle to internalize the multi-step logic due to their constrained ability. Instead, small models perform better when fine-tuned on shorter, simpler reasoning chains that align more closely with their intrinsic learning capacity. This suggests that small models struggle to process overly elaborate reasoning traces or adapt to the distribution shifts introduced by stronger teachers, ultimately limiting their ability to generalize effectively. To address the challenge described above, we propose Mix Distillation , a simple yet effective approach that balances reasoning complexity by blending different types of reasoning traces. Specifically, our method comprises two configurations: (1) Mix-Long – A combination of long and short CoT examples, ensuring that small models are exposed to both detailed and concise reasoning steps. (2) Mix-Large – A mixture of responses from both larger and smaller models, allowing small models to learn from reasoning chains that are better suited to their capacity. Our experiments demonstrate that Mix Distillation consistently improves small model reasoning performance compared to standard distillation. For instance, Qwen2.5-3B-Instruct improves by more than 8 points on MATH and AMC using Mix-Long, compared to direct training on long CoT data. Qwen2.5-3B-Instruct gains more than 7 points on MATH, AIME and AMC using Mix-Large compared with training on large teacher CoT data. These findings highlight a fundamental limitation of direct strong model distillation and emphasize the importance of adapting reasoning complexity for effective knowledge transfer. By carefully designing fine-tuning strategies, we provide new insights into overcoming the constraints of small model learning, making them more effective at reasoning-intensive tasks. 2 Preliminaries 2.1 Notation Let x=(x_{1},x_{2},\dots,x_{n}) represent an input sequence (e.g., a prompt), and y=(y_{1},y_{2},\dots,y_{m}) be the corresponding output sequence. We consider a LLM parameterized by \theta , which predicts the next token following a conditional distribution \pi_{\theta}\bigl(y_{t}|x,y_{1:t-1}\bigr) . We denote by \text{CoT}(y)\subseteq y the subset of tokens in the generated output that encodes a chain-of-thought , often serving as a reasoning trace or explanatory sequence. Throughout this work, we use the term short CoT , to describe concise reasoning paths to arrive at solutions ( Min et al., 2024 ; Yeo et al., 2025 ) and long CoT to describe an extended reasoning sequence that is not only longer but also demonstrates more complex reflective thoughts ( Qwen, 2024b ; Yeo et al., 2025 ) . Additionally, we use the term large teacher CoT to refer to the reasoning trace generated by a larger teacher model, and the term small teacher CoT for the reasoning steps produced by a smaller teacher model. Please see Appendix D for more examples. 2.2 Supervised Fine-Tuning (SFT) Supervised fine-tuning (SFT) is widely adopted to enhance reasoning capabilities of LLMs on a dataset \mathcal{D}=\{(x^{i},y^{i})\}_{i=1}^{N} , where y^{i} can be short CoT, long CoT, strong model CoT or weak model CoT sequences. The SFT process updates the parameters \theta of a language model by minimization the negative log-likelihood loss over the instruction dataset \mathcal{D} . 3 Small Model Learnability Gap In this section, we fine-tune student models using different CoT data. We then reveal the small model learnability gap given the performance of fine-tuned models. 3.1 Experiment Setup Datasets. We use the 7,500 prompt set of MATH ( Hendrycks et al., 2021 ) . This dataset encompasses seven math topics such as advanced calculus, geometry, and linear algebra. Student models. Our study considers ten student models from the Qwen ( Qwen, 2024a ) and Llama ( Meta, 2024a ; Meta, 2024b ) model families of varying sizes. These models include the Instruct version of Qwen2.5-0.5B , Qwen2.5-1.5B , Qwen2.5-3B , Qwen2.5-7B , Qwen2.5-14B , and Qwen2.5-32B , and the Instruct version of Llama3.2-1B , Llama3.2-3B , Llama3.1-8B , and Llama3.3-70B . A comprehensive overview of the student models is presented in Table 4 of Appendix A . Teacher models. To compare long CoT with short CoT, we use QwQ-32B-Preview ( Qwen, 2024b ) to generate long CoT sequences and Qwen2.5-32B-Instruct as the response generator for short CoT. Within each model family, we designate the larger scale model as the large teacher and the smaller scale model as the small teacher. This includes Qwen2.5-72B-Instruct vs Qwen2.5-3B-Instruct , Llama3.1-70B-Instruct vs Llama3.1-8B-Instruct , and Gemma2-27B-it vs Gemma2-9B-it . Figure 2: Long CoT Gap ( \Delta_{Long}=P_{Long}-P_{Short} ) of student models with different models sizes