Self-Distilled RLVR

Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, Nan Duan · 2026

Read Self-Distilled RLVR by Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, Nan Duan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03128.txt.

Introduction

Yang and colleagues untangle when self-distillation helps reinforcement learning with verifiable rewards, a setup where a model teaches itself using privileged answers. They show signals drawn only from that privileged teacher leak information and destabilize long training. Their fix, RLSD, uses self-distillation only to size each update, while verifiable rewards set the direction, lifting the training ceiling and steadying learning.

Catalog record

CREATORChenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, Nan Duan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79833
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Distilled RLVR Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, Nan Duan Abstract On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose RLSD ( RL VR with S elf- D istillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability. 1]Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2]School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3]JD.COM \contribution [*]Equal contribution. \contribution [†]Corresponding author. \checkdata [Email]; ; Figure 1 : Performance on the trained Qwen3-VL-8B-Instruct model. In (a), OPSD reaches its peak performance early and degrades, whereas RLSD inherits the training stability of GRPO while achieving a higher convergence ceiling. In (b), GRPO and RLSD report results at 200 training steps, while GRPO (2 \times steps) reports results at 400 steps; RLSD at 200 steps already surpasses GRPO trained for twice as many steps, demonstrating faster convergence. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) methods such as GRPO [ 1 ] have become a central paradigm for training large reasoning models [ 2 , 3 ] , where each trajectory receives only a single scalar signal determined by the outcome from the environment. On-policy distillation (OPD) [ 4 , 5 ] complements this by leveraging a stronger teacher model to provide dense, token-level logits as learning signals along the student’s own sampled trajectories, enriching the trajectory-level supervision to the token level and thereby achieving faster convergence. Recent work has shown that OPD from advanced teachers can match or even outperform RLVR [ 6 ] , establishing it as an equally compelling paradigm (see Table 1 for a systematic comparison). Despite its effectiveness, OPD relies on a separate, typically much larger teacher model, incurring substantial computational overhead. Moreover, since OPD requires computing token-level distributions over a shared vocabulary, the teacher and student models must share the same vocabulary, significantly reducing the practical scalability of this paradigm. On-policy self-distillation (OPSD) [ 7 , 8 ] offers an appealing alternative: a single model serves as both teacher and student, where the teacher is the same model conditioned on privileged information r (such as a verified reasoning trace or environmental feedback) and the student operates solely on the input query. OPSD achieves several-fold improvements in token efficiency over GRPO without requiring any external model. However, we prove this efficiency gain is fragile: as illustrated by the red curve in Figure 1 (a), performance peaks early and subsequently deteriorates, accompanied by systematic privileged information leakage, where the model explicitly appeals to an invisible reference solution during inference despite never having access to it (see Figure 2 for a representative example). These phenomena raise a natural question: why does OPD work while OPSD fails? We trace the answer to a structural distinction between the two settings. In OPD, teacher and student observe the same input (information-symmetric), so the teacher’s dense signal reflects superior reasoning under shared information access. In OPSD, the teacher conditions on privileged information that the student cannot observe (information-asymmetric), creating a fundamental mismatch. We prove that this asymmetry renders the OPSD objective ill-posed : it contains an irreducible mutual information gap I(Y_{t};R\mid X,Y_{<t})>0 that the student can never eliminate, regardless of its capacity (Theorem 1 ). At the gradient level, we show that while the expected OPSD gradient is benign, the per-sample gradients carry an r -specific deviation whose variance is proportional to this mutual information. These theoretical findings are directly corroborated by our diagnostic experiments: the on-policy KL divergence between teacher and student stagnates under OPSD with no sustained reduction, whereas the same metric decreases steadily under OPD, as shown in Figure 3 . Early in training, the beneficial gradient component dominates, producing rapid improvement; as the student approaches the teacher’s marginal distribution, the deviation takes over, and its path-dependent accumulation drives the model toward encoding x\to r correlations in its parameters. This two-phase dynamic precisely accounts for the observed pattern of early gains followed by progressive degradation. Table 1 : Comparison of training paradigms for post-training LLMs. Method Trajectory Efficiency Leakage Risk Signal Direction Anchoring SFT ✗ Off-policy ✓ High ✓ N/A ✓ Rich ✗ Teacher RLVR (GRPO) ✓ On-policy ✓ High ✓ N/A ✗ Weak ✓ Environment OPD ✓ On-policy ✗ Low ✓ N/A ✓ Rich ✗ Teacher OPSD ✓ On-policy ✓ High ✗ Severe ✓ Rich ✗ Teacher RLSD (Ours) ✓ On-policy ✓ High ✓ N/A ✓ Rich ✓ Environment Our analysis pinpoints the root cause: in all distribution-matching formulations, the teacher’s privileged evaluation P_{T}(y_{t}\mid r) enters the gradient direction , making leakage structurally unavoidable regardless of how the distillation target is compressed. Yet the evidence ratio P_{T}(y_{t})/P_{S}(y_{t}) also carries a useful signal: it measures how much the privileged information revises the model’s belief about each token. The challenge is therefore not to discard this signal, but to change how it is used. A key insight underlying our design is that the signals governing update direction and update magnitude have asymmetric requirements: the directional signal can be sparse but must be reliable , as an erroneous direction harms the policy; the magnitude signal, by contrast, benefits from being as dense as possible to enable fine-grained discrimination among tokens . We propose RL VR with S elf- D istillation ( RLSD ), which instantiates this principle by repurposing the teacher from a generative target to a magnitude evaluator . Specifically, the environment reward determines the direction of each token’s update (reinforcement or penalization), while the teacher’s evidence ratio modulates only the magnitude . This decoupling anchors gradient directions exclusively to the reliable environment reward, while retaining the teacher’s dense, token-level assessment for fine-grained credit discrimination across token positions. Notably, prior approaches to token-level credit assignment, such as value-function estimation in PPO [ 9 ] and various credit assignment methods [ 10 , 11 ] , typically require training auxiliary networks or incur substantial additional overhead, yet still produce noisy estimates. In contrast, self-distillation provides a natural and virtually cost-free source of per-token credit information, requiring only a single additional forward pass. As summarized in Table 1 , RLSD is the only paradigm that simultaneously achieves on-policy training, high token efficiency, rich update signals, and environment-anchored optimization, while serving as a drop-in replacement for the uniform advantage in standard GRPO without any auxiliary loss or model. Our

Read the complete text