Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation

Yuwei Zhang, Sha Li, Changlong Yu, Qin Lu, Shuowei Jin, Chengyu Dong, Haoran Liu, Ilgee Hong, Xintong Li, Zhenyu Shi, Bing Yin, Jingbo Shang, UC San Diego, Amazon, Georgia Institute of Technology {yuz163,jshang}@ucsd.edu · 2026

Read Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation by Yuwei Zhang, Sha Li, Changlong Yu, Qin Lu, Shuowei Jin, Chengyu Dong, Haoran Liu, Ilgee Hong, Xintong Li, Zhenyu Shi, Bing Yin, Jingbo Shang, UC San Diego, Amazon, Georgia Institute of Technology {yuz163,jshang}@ucsd.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12741.txt.

Introduction

Zhang and eleven colleagues help language models improve from world interactions even when success is rare. Self-distillation offers a good route, yet existing methods treat world feedback as a passive signal and rely heavily on winning demos. Their RESD setup transforms raw failure feedback into active fixing help. It reads failed runs, making backward-looking notes to spot local errors, and keeps a global playbook of reused lessons across training steps.

Catalog record

CREATORYuwei Zhang, Sha Li, Changlong Yu, Qin Lu, Shuowei Jin, Chengyu Dong, Haoran Liu, Ilgee Hong, Xintong Li, Zhenyu Shi, Bing Yin, Jingbo Shang, UC San Diego, Amazon, Georgia Institute of Technology {yuz163,jshang}@ucsd.edu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49671
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation Yuwei Zhang, Sha Li, Changlong Yu, Qin Lu, Shuowei Jin, Chengyu Dong, Haoran Liu, Ilgee Hong, Xintong Li, Zhenyu Shi, Bing Yin, Jingbo Shang, UC San Diego, Amazon, Georgia Institute of Technology {yuz163,jshang}@ucsd.edu Abstract Enabling Large Language Models (LLMs) to continuously improve from environmental interactions is a central challenge in post-training. While on-policy self-distillation offers a promising paradigm, existing methods predominantly treat environmental feedback as a passive conditioning signal. Consequently, they heavily rely on successful demonstrations and struggle to learn in rare-success regimes. To bridge this gap, we introduce Reflection-Enhanced Self-Distillation ( Resd ), a framework that transforms raw failure feedback into an active source of corrective supervision. Instead of passively appending feedback, Resd interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook to preserve reusable lessons across training steps. The enriched context enables the self-teacher to provide actionable token-level supervision even in the absence of successful rollouts. Empirical evaluations on multiple continual learning tasks demonstrate that Resd substantially outperforms standard self-distillation baselines. Furthermore, Resd achieves significantly faster early-stage improvement than GRPO with 8 × 8\times samples using only a single rollout per prompt, highlighting its superior interaction efficiency. Project Page GitHub 1 Introduction Figure 1 : Resd improves interaction efficiency during training. The x-axis is the number of samples. A fundamental challenge in the post-training of Large Language Models (LLMs) is enabling continuous improvement through environmental interactions. Traditionally, Reinforcement Learning (RL) algorithms, such as PPO or GRPO, have been the standard paradigm for aligning models with desired outcomes. However, a critical bottleneck in deploying RL for complex, multi-step tasks is the reliance on sparse reward signals. In scenarios where successful trajectories are exceedingly rare, standard RL often struggles to effectively bootstrap policies because it lacks dense supervision to guide the model through a vast exploration space. To mitigate the sparse reward problem, On-Policy Distillation (OPD) has emerged as a promising alternative ( 1 ; 12 ; 23 ) . Rather than relying on a single scalar reward at the end of a rollout, OPD leverages a stronger or privileged teacher model to compute target probabilities for every single token in the generated trajectory. This mechanism elegantly translates sparse trajectory-level outcomes into dense, token-level learning signals. However, standard OPD requires maintaining a separate expert model, which incurs high computational costs and risks distribution mismatch between the teacher and the student. To address this, self-distillation variants like SDPO instantiate the teacher from the model itself ( 6 ; 30 ) . By conditioning the self-teacher on environmental feedback, SDPO eliminates the need for an external oracle while perfectly aligning the supervisory signal with the model’s own generation distribution. While self-distillation resolves both reward sparsity and distribution mismatch, its success hinges entirely on the quality of the self-teacher’s supervision. Current methods predominantly treat environmental feedback as a static, passive conditioning variable. We argue that this overlooks a key design choice: the structural representation of the feedback itself. In sparse-reward regimes where successful demonstrations are absent, the teacher is forced to derive supervision almost entirely from failed rollouts. We demonstrate in Figure 1 that simply feeding raw failure outcomes or unstructured environment errors into the teacher context yields only marginal student improvements. Motivated by this observation, we propose Reflection-Enhanced Self-Distillation ( Resd ), a framework that enriches the teacher context with actively interpreted feedback. For each failed rollout, Resd first generates a retrospective reflection that identifies the likely cause of failure and the correction that would have avoided it. It then organizes recurring lessons into a persistent playbook, allowing future teacher prompts to reuse feedback-derived knowledge across training steps. The resulting enriched teacher context transforms raw failure feedback from a passive diagnostic signal into an actionable and reusable source of corrective supervision, enabling self-distillation to improve even when successful demonstrations are unavailable. Empirically, we evaluate Resd on tasks designed to reflect continuous learning beyond common post-training distributions. These tasks are novel to the model, often begin in rare-success regimes, and provide rich execution feedback despite receiving sparse binary rewards. Across these settings, Resd substantially improves over standard self-distillation baselines and achieves faster early-stage improvement than GRPO while using only a single rollout per prompt. These results suggest that structured feedback enables sample-efficient bootstrapping from failures, while reward-based optimization remains a complementary tool once sufficient successful rollouts are available. Our contributions are as follows: • We identify feedback formulation as a critical design axis for on-policy self-distillation especially in sparse-reward regimes. • We propose Reflection-Enhanced Self-Distillation ( Resd ), which enriches the teacher context with retrospective reflections and a persistent playbook. By interpreting failures and preserving recurring lessons across training steps, Resd turns raw feedback from a passive diagnostic signal into reusable corrective supervision. Empirically, we characterize the sample-efficiency of feedback-enhanced self-distillation under an online streaming protocol. • More broadly, by revealing that self-improving models are highly sensitive to how feedback is structured, we position our reflection and playbook mechanisms as a plug-and-play module that can readily enhance other self-distillation objectives. 2 Preliminaries Recent advancements such as Self-Distillation Policy Optimization (SDPO) ( 6 ) and On-Policy Self-Distillation (OPSD) ( 30 ) introduce a “self-teacher” paradigm that replaces sparse outcomes in Reinforcement Learning with dense, token-level supervisions without the need for a massive separate teacher model. In these frameworks, a “self-teacher” policy is typically instantiated as the model’s current or moving-average weights \theta_{\text{old}} , operating in a privileged context. The teacher is conditioned on both the intermediate state s_{t}=(x_{t},y_{<t}) and the retrospective feedback c(x,y) , producing a corrected, feedback-informed distribution \pi_{\theta_{\text{old}}}(\cdot|x,y_{<t},c) . The base policy \pi_{\theta} is then optimized to match this privileged distribution at every decoding step. We formulate this matching objective universally using f-divergences . For a given time step t , let the likelihood ratio for any token v\in\mathcal{V} be defined as \tau^{t}_{v}=\frac{\pi_{\theta_{\text{old}}}(v|x,y_{<t},c)}{\pi_{\theta}(v|x,y_{<t})} . To prevent extreme ratios from destabilizing training, we clip the likelihood ratio to a bounded range: \tilde{\tau}^{t}_{v}=\operatorname{clip}\!\left(\tau^{t}_{v},\;\epsilon_{\min},\;\epsilon_{\max}\right) The unified self-distillation objective then minimizes the expected f-divergence across all tokens t\in[1,T] : \mathcal{L}_{\text{SD}}(\theta)=\mathbb{E}_{x,y\sim\pi_{\theta},f}\left[\sum_{t=1}^{T}\sum_{v\in\mathcal{V}}\pi_{\theta}(v|x,y_{<t})\cdot f(\tilde{\tau}^{t}_{v})\right] (1) where f(\tau) is a convex function. By selecting different f , we can recover various divergence measures, such as forward KL, reverse KL and Jensen-Shannon Divergence (JSD), each offering unique optimization properties. We provide a detailed discussion on the choice of f and its implications in Appendix A . Notice that, since we sample trajectories from the base policy \pi_{\theta} , the self-distillation objective is inherently on-policy. Such a formulation allows the model to recover forgotten behaviors after midtraining ( 17 ; 3 ; 18 ; 10 ) and prevent from exposure bias ( 12 ; 2 ) . 3 Method Figure 2 : Overview of the Resd framework. The student generates a rollout and receives environment feedback. On failure, a local self-reflection diagnoses the error and a global playbook curation step distills reusable lessons into a persistent playbook. The enriched context (reflection, curated playbook, and any cached solutions) is fed to the teacher prompt, whose output distribution provides token-level supervision to the student via a KL loss. Our work builds on the finding that existing self-distillation methods rely heavily on successful demonstrations, and their supervision

Read the complete text