Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning

Shuyao Xu Cheng Peng Jiangxuan Long Weidi Xu Wei Chu Yuan Qi · 2025

Read Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning by Shuyao Xu Cheng Peng Jiangxuan Long Weidi Xu Wei Chu Yuan Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24850.txt.

Introduction

Shuyao Xu and co-authors rescue the wrong answers that teacher models produce in this 2025 paper. When a big reasoning model teaches a small one, its failed attempts are normally thrown away. The team asks whether those failures still carry useful signal for offline training.

Their recipe first fine-tunes the student on correct traces only, then refines it using both right and wrong traces. A simple reinforcement-style objective they call REDI beat popular preference methods at this second stage. A small model trained on 131 thousand traces matched a rival trained on 800 thousand private ones. The lesson: discarded mistakes can be remarkably efficient teaching material.

Catalog record

CREATORShuyao Xu Cheng Peng Jiangxuan Long Weidi Xu Wei Chu Yuan Qi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24850

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning Shuyao Xu Cheng Peng Jiangxuan Long Weidi Xu Wei Chu Yuan Qi Abstract Recent advances in model distillation show that data from advanced reasoning models can effectively train smaller student models. However, standard practices discard incorrect reasoning traces—valuable, yet underutilized data. This paper addresses the critical question: How can both positive and negative distilled reasoning traces be effectively leveraged to maximize LLM reasoning performance in an offline setting? We employ a two-stage training recipe: first, Supervised Fine-Tuning (SFT) on positive traces, followed by a refinement stage using both positive and negative traces. We find that a simple REINFORCE-style objective, which we term the Reinforcement Distillation (REDI) objective, outperforms established preference optimization methods like DPO and SimPO in this distillation context. Our empirical evaluations demonstrate the effectiveness of this approach. Notably, our Qwen-REDI-1.5B model, trained on just 131k traces from the open Open-R1 dataset, achieves an 83.1% score on MATH-500. Its performance matches that of DeepSeek-R1-Distill-Qwen-1.5B , a model trained on 800k proprietary data. This result showcases the remarkable data efficiency of utilizing previously discarded negative traces. 1 Introduction Recent breakthroughs with large reasoning models, such as DeepSeek-R1 and OpenAI’s o1, have demonstrated remarkable capabilities in complex reasoning tasks ( DeepSeek-AI et al., 2025 ; OpenAI et al., 2024 ) . Techniques like test-time scaling facilitate longer Chain-of-Thought (CoT) processes and induce sophisticated reasoning behaviors, enhancing model performance in domains like mathematics. For base models initially lacking such advanced reasoning, two primary methods are employed to cultivate these abilities. The first, large-scale reinforcement learning (RL), directly applies RL algorithms to the base model, continually optimizing it through online exploration ( DeepSeek-AI et al., 2025 ; Pan et al., 2025 ; Zeng et al., 2025 ) . However, RL approaches typically demand strong base models to achieve their full potential and are computationally intensive ( Yue et al., 2025 ; DeepSeek-AI et al., 2025 ) . In contrast, distillation—learning from reasoning traces (e.g., CoT) generated by large “teacher“ models—emerges as an attractive alternative for smaller, more efficient student models. This approach offers a practical and cost-effective pathway to extend their reasoning capabilities ( Team, 2025 ; DeepSeek-AI et al., 2025 ) . Benefiting from open datasets distilled from powerful reasoning models like DeepSeek-R1 ( Team, 2025 ; Labs, 2025 ; Face, 2025 ) , openly post-trained models have shown strong performance ( Team, 2025 ; Ye et al., 2025 ; Muennighoff et al., 2025 ; Wen et al., 2025 ) , although a performance gap remains compared to their closed-data counterparts. Figure 1: Standard distillation practices via Rejection Sampling vs. our proposed Reinforcement Distillation (REDI). Our REDI recipe can utilize previously discarded incorrect reasoning traces generated by the teacher and yield stronger distilled models. Table 1: Model Performance Comparison (pass@1 over 16 samples) across reasoning benchmarks. Our Qwen-REDI-1.5B , trained with the REDI recipe on just 131k open data points, achieves the highest average score. This performance surpasses DeepSeek-R1-Distill-Qwen-1.5B (trained on 800k proprietary data) ( DeepSeek-AI et al., 2025 ) , demonstrating REDI’s remarkable data efficiency . REDI enhances reasoning by effectively utilizing both positive and negative distilled examples. Values in bold indicate the best performance in each column. * Officially reported pass@1 results. Model MATH-500 AIME24 AMC23 Minerva OlympiadBench Avg. Qwen2.5-Math-1.5B-Instruct * 75.8 10.0 60.0 29.4 38.1 42.7 DeepSeek-R1-Distill-Qwen-1.5B 83.2 \pm 0.2 28.3 \pm 1.0 62.1 \pm 0.8 26.0 \pm 0.4 43.1 \pm 0.3 48.6 \pm 0.3 Qwen-SFT-1.5B-5ep (SFT Baseline) 80.4 \pm 0.3 21.9 \pm 1.1 57.5 \pm 0.6 27.5 \pm 0.4 41.5 \pm 0.3 45.8 \pm 0.3 Qwen-REDI-1.5B 83.1 \pm 0.3 28.1 \pm 1.1 62.4 \pm 0.6 28.8 \pm 0.3 45.2 \pm 0.2 49.5 \pm 0.3 However, current distillation methodologies predominantly rely on rejection sampling, which involves leveraging only positive 1 1 1 We use “positive” interchangeably with “correct”, and “negative” interchangeably with “incorrect”. reasoning traces—those whose final answers are verified. This practice means that negative traces, despite the significant computational effort invested in their generation, are typically underutilized. We hypothesize that these negative traces contain vital insights into common pitfalls and nuanced errors from which smaller models could learn, thereby further unlocking the potential of distilled data. This leads to the central research question we address: How can we effectively leverage both positive and negative distilled reasoning traces to maximize LLM reasoning performance with a fixed distilled open dataset? To address this challenge, we first investigate the application of established preference optimization methods, such as Direct Preference Optimization (DPO) ( Rafailov et al., 2024 ) and SimPO ( Meng et al., 2024 ) , to this offline distillation setting. Our analysis reveals a critical performance-stability trade-off : while the Kullback-Leibler (KL) divergence penalty \beta is essential for stable training, it simultaneously constrains the model’s peak achievable performance 2 2 2 Performance refers to the test-time accuracy of the best checkpoint in a training run. . This discovery motivated a deeper analysis, where we found that in the \beta\to 0 limit, these complex objectives converge to a simpler, reference-free and REINFORCE-style objective. This insight forms the basis of our approach. We propose Re inforcement Di stillation (REDI), which adopts this performant but unstable objective and introduces a asymmetric weighting scheme. By down-weighting the gradient from negative traces, REDI restores training stability without sacrificing the performance benefits, offering a simple and effective method for learning from both positive and negative signals. Our key contributions are: 1. We provide the first systematic study on the utilization of both correct and incorrect distilled reasoning traces. We identify and analyze a performance-stability trade-off inherent in the KL regularization of methods like DPO, demonstrating that it limits peak performance in this practical setting. 2. Motivated by this analysis, we propose the Reinforcement Distillation (REDI) objective , a simple, asymmetrically weighted, and REINFORCE-style loss function. REDI is designed to capture the high-performance potential of the \beta\to 0 limit of DPO while mitigating the associated training instability, providing a simpler and more effective alternative. 3. We empirically demonstrate that our two-stage recipe of SFT combined with REDI training consistently outperforms both Rejection Sampling SFT and SFT combined with DPO/SimPO. Our Qwen-REDI-1.5B model achieves performance comparable to models trained on much larger proprietary datasets, showcasing the data efficiency of our recipe. The remainder of this paper is organized as follows: Section 2 details the Reinforcement Distillation methodology. Section 3 describes the experimental setup. Section 4 presents our results and analysis. Section 5 discusses related work, and Section 6 concludes the paper. 2 Methodology 2.1 Problem Setting and Data We operate in an offline distillation setting with a fixed dataset collected via a common distillation pipeline. The dataset originates from a set of problems, each denoted by x . For each problem x , a capable “teacher" model is employed to generate reasoning traces. The generation process for a specific problem x continues until a correct reasoning trace, y_{w} , is successfully produced. During these attempts, incorrect traces, y_{l} , might also be generated before y_{w} is obtained. From these generated traces, we construct two distinct datasets for our two-stage training recipe: 1. Positive Traces Dataset ( \mathcal{D}_{\text{SFT}} ): This dataset comprises all pairs (x,y_{w}) , where y_{w}

Read the complete text