Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation

Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta, Dawei Li, Zhao Yang · 2026

Read Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation by Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta, Dawei Li, Zhao Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09253.txt.

Introduction

Jiang and four colleagues investigate high loss tokens in on policy distillation, where a student learns from a teacher using paths sampled from its own policy. High loss tokens give the clearest sign of student teacher mismatch under distillation's per token KL objective, which existing studies expect to fade as training converges. The team's analysis shows otherwise: even after training saturates, a substantial subset keeps exhibiting persistently high loss.

These tokens, termed Rock Tokens, can make up to 18 percent of generated outputs. Two paradoxes: despite their high frequency driving much of the total gradient signal, Rock Tokens themselves stay stagnant, resisting teacher driven corrections. Causal intervention shows these tokens add almost nothing to the model's actual reasoning performance. The findings suggest vast optimisation bandwidth goes to structural and discourse residuals the student need not internalise, and bypassing these stumbling blocks can streamline alignment and challenge uniform token weighting.

Catalog record

CREATORYuxuan Jiang, Runchao Li, Shubhashis Roy Dipta, Dawei Li, Zhao Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta, Dawei Li, Zhao Yang Abstract While recent work in Reinforcement Learning with Verifiable Rewards (RLVR) has shown that a small subset of critical tokens disproportionately drives reasoning gains, an analogous token-level understanding of On-Policy Distillation (OPD) remains largely unexplored. In this work, we investigate high-loss tokens, a token type that—as the most direct signal of student-teacher mismatch under OPD’s per-token KL objective—should progressively diminish as training converges according to existing studies; however, our empirical analysis shows otherwise. Even after OPD training reaches apparent saturation, a substantial subset of tokens continues to exhibit persistently high loss; these tokens, which we term Rock Tokens, can account for up to 18% of the tokens in generated outputs. Our investigation reveals two startling paradoxes. First, despite their high occurrence frequency providing a disproportionately large share of total gradient norms, Rock Tokens themselves remain stagnant throughout training, resisting teacher-driven corrections. Second, through causal intervention, we find that these tokens provide negligible functional contribution to the model’s actual reasoning performance. These findings suggest that a vast amount of optimization bandwidth is spent on structural and discourse residuals that the student model cannot or need not internalize. By deconstructing these dynamics, we demonstrate that strategically bypassing these "stumbling blocks" can significantly streamline the alignment process, challenging the necessity of uniform token weighting and offering a more efficient paradigm for large-scale model distillation. Code is available at: https://github.com/YuxuanJiang1/Rock-Token 1 Introduction On-Policy Distillation (OPD) has established itself as a cornerstone of the modern post-training pipeline for Large Language Models (LLMs) [ 1 ] . By aligning a student model with a superior teacher through trajectories sampled from its own policy, OPD enables a token level of reasoning refinement that goes beyond static Supervised Fine-Tuning (SFT) [ 68 ] . This effectiveness has been validated by industrial works such as DeepSeek-V4 [ 13 ] , MiMo [ 76 ] , and Qwen-3 [ 82 ] , where OPD serves as a vital component alongside SFT or Reinforcement Learning with Verifiable Rewards (RLVR) to further squeeze out reasoning performance. Recent studies in Reinforcement Learning with Verifiable Rewards (RLVR) [ 15 , 16 , 22 ] have revealed that not all tokens contribute equally to model learning: a small subset of critical tokens, such as high-entropy "forking tokens" at decisive branching points, disproportionately drives reasoning gains [ 71 ] . While this token-level perspective has reshaped our understanding of RLVR dynamics, an analogous investigation in OPD remains largely unexplored, despite OPD’s reliance on dense token-level supervision. In this work, we initiate such an investigation by focusing on a token type uniquely meaningful to the OPD paradigm: high-loss tokens. Unlike RLVR, where token importance is inferred from policy entropy or reward signals, OPD provides a direct measure of student-teacher mismatch through per-token KL divergence; high-loss tokens thus explicitly mark the positions where the student’s policy diverges most sharply from the teacher’s, making them natural candidates for the “critical correction points” that drive alignment [ 63 ] . Intuitively, as the student aligns with the teacher during the OPD process, both the magnitude and cardinality of these high-loss tokens are expected to diminish, eventually leading to an optimization plateau. However, our empirical investigations reveal a striking phenomenon: even after training has reached apparent saturation, a stubborn population of tokens—which we formally define as Rock Tokens—continues to exhibit consistently high loss. Notably, while these tokens represent a small minority of the vocabulary (approximately 6%), they account for as much as 18% of the total token frequency in the model’s output. This persistence challenges conventional understanding of model convergence and raises a fundamental question: Are Rock Tokens indispensable Pillars for policy alignment, or are they Stumbling Blocks that create unnecessary optimization redundancy? Figure 1: The lifecycle and functional impact of Rock Tokens in OPD. (a) Phenomenon : Identification of optimization-resistant tokens. (b) Mechanism : Causal evidence of structural redundancy via token knock-out. (c) Utility : Performance parity achieved through strategic gradient sparsification. To provide a systematic understanding of the Rock Token phenomenon, our research is structured across three investigative dimensions: identifying what are rock tokens, analyzing why they emerge and persist, and uncovering how they collectively impact the overall OPD training process. This comprehensive framework is operationalized through the following three stages: First, we characterize the identity of Rock Tokens (What). We begin by identifying Rock Tokens linguistic and statistical properties, revealing that they primarily comprise syntactic and structural scaffolding —such as formatting delimiters, whitespace symbols, and high-frequency discourse markers (e.g., “So”, “Wait”). To determine the dynamic role of these elements, we employ quantitative gradient dynamics analysis to address RQ1: Do Rock Tokens still provide useful learning signals during optimization? By examining whether their persistent high loss effectively translates into constructive gradients, we assess whether these tokens offer steady directional guidance or merely represent stagnant optimization noise throughout the OPD process. Second, we investigate the underlying mechanisms behind their persistence (Why). Beyond the actual contribution of Rock Tokens during training is clarified, a more profound question emerges: why do these tokens persist in the first place? We hypothesize that the student model develops a strong decoding dependency on these specific tokens to maintain its reasoning flow, creating a path dependency that resists corrections from the teacher model. This leads to our RQ2: Does Rock Token persistence stem from student path dependency? To investigate this, we conduct token knock-out experiments as a form of ablation analysis, examining whether the student’s logical coherence is so deeply intertwined with these persistent residuals that it becomes structurally unable to shift its policy toward the teacher’s distribution. Finally, we evaluate how Rock Tokens influence the overall OPD training process (How). Finally, we arrive at a provocative question: what would happen if these tokens were excluded from the very beginning of OPD training? This leads to our final investigation, RQ3: What is the genuine functional contribution of Rock Tokens to model training? By freezing the gradients of these tokens from the onset of training, we intuitively isolate their functional necessity throughout the entire distillation process. This causal intervention allows us to rigorously determine whether Rock Tokens serve as indispensable pillars of the model’s reasoning integrity or if they represent a strategic opportunity to bypass computational redundancy and optimize the efficiency of the alignment pipeline. In summary, our key contributions are as follows: • For the first time, we identify and characterize Rock Tokens , a persistent high-loss phenomenon that unveils a previously hidden aspect of optimization dynamics in On-Policy Distillation. • We deconstruct the intricate dynamics of the OPD process by systematically uncovering the functional roles of Rock Tokens across both the training and inference phases. • Our research reveals a critical inefficiency in current alignment pipelines, demonstrating that a substantial portion of training signals in OPD provides significantly lower-than-expected contributions to the model’s ultimate reasoning performance. 2 Rock Tokens Figure 2: Empirical identification and stability of Rock Tokens. (a) Per-token KL \widehat{\ell}_{v} vs. frequency on N{=}500 MATH-500 trajectories: rare tokens are noise-dominated, while the Rock Score R(v) isolates true Rock Tokens (red) at the upper edge of stable frequency bands. (b) Per-sequence Rock-Token

Read the complete text