Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Hongxia Yang · 2026
Read Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation by Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Hongxia Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26844.txt.
Introduction
Wang and eight colleagues ask which teacher signals in on-policy distillation actually teach the student. Raw disagreement between teacher and student mixes learnable gaps, where the teacher backs the student's own top candidates, with incompatible gaps, where teacher mass sits outside student support. Their token teachability score separates the two and predicts gains better than raw divergence. A teachability-aware method then beats full-token training while keeping only 5 percent of tokens.
Catalog record
| CREATOR | Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Hongxia Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51498 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Hongxia Yang Abstract On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which token-level teacher signals are actually learnable? Using a fixed-context diagnostic that measures same-context teacher–student KL reduction, we show that raw KL disagreement is a coarse proxy for learning value. It conflates learnable disagreement , where the teacher assigns corrective mass to the student’s top- K K candidates, with incompatible disagreement , where the teacher places mass mostly off the student’s current support. We formalize this local compatibility as token teachability and show that it better predicts fixed-context improvement than raw KL alone. Motivated by this finding, we propose Teachability-Aware OPD (TA-OPD), a lightweight token-position selection method that applies OPD loss to high-teachability positions without reward models or verifiers. Across Qwen2.5 and Qwen 3 teacher–student settings, TA-OPD often surpasses full-token OPD with only 5% retained tokens and improves over entropy- and divergence-based baselines. Our results reframe selective OPD as selecting learnable teacher signals rather than merely salient tokens. 1 Introduction Knowledge distillation has long been used to transfer the behavior of a large teacher model to a smaller student model ( 12 ; 38 ; 4 ) . For large language models (LLMs), a particularly effective variant is on-policy distillation (OPD), which trains a student policy on its own rollouts using token-level supervision from a teacher policy ( 1 ; 8 ; 31 ) . Compared with off-policy distillation ( 37 ; 24 ) , OPD supervises the student on states that it actually visits, reducing distribution mismatch between training and deployment. However, this also exposes a central challenge: the teacher provides dense supervision at every token, but not every token-level teacher signal is equally useful. Figure 1: Token teachability. A: Low-entropy, high-KL tokens can contain learnable disagreement D^{L} , which stays within the student’s local support, and incompatible disagreement D^{I} , which shifts off support. B: TA-OPD computes disagreement D_{t} and compatibility C_{t} , ranks positions by s_{t}^{\mathrm{teach}}=\tilde{D}_{t}\tilde{C}_{t} , and keeps only high-teachability OPD supervision. Recent work shows that OPD supervision is highly non-uniform ( 13 ; 20 ) . Selective OPD therefore allocates token budget to salient positions, such as high-entropy tokens or low-entropy tokens with large teacher–student disagreement ( 10 ; 26 ; 27 ) . However, these criteria measure uncertainty or disagreement, not the learning effect of the induced token-level supervision. In on-policy training, two tokens with similar KL disagreement can lead to very different updates, depending on how the teacher distribution interacts with the student’s current predictive state (Fig 1 .A). This raises a basic question: which token-level teacher signals in OPD are actually learnable? To answer this question, we need to separate token-level learning value from the rollout in which a token appears. End-to-end OPD performance entangles token supervision with sampling noise, downstream context shifts, and interactions across positions. We therefore introduce a fixed-context diagnostic : we collect student-generated prefixes, freeze them as a context bank, and rescore both the initial and trained students against the same teacher distribution. For each token, we measure the same-context reduction in teacher–student KL and ask which pre-update disagreement signals predict this local improvement. This diagnostic allows us to study whether a token-level teacher signal is aligned with useful learning, rather than merely salient under entropy or raw KL. Our diagnostic shows that high KL disagreement is not a uniform learning signal. Even in the low-entropy, high-divergence regime targeted by prior selectors, raw disagreement mixes two qualitatively different cases: In learnable disagreement , the teacher assigns corrective mass to the student’s top- K candidates, yielding high student-support coverage and better alignment with KL-disagreement reduction. In incompatible disagreement , teacher mass falls mostly outside the student’s current support, producing a large KL gap but weak local improvement. We refer to this local compatibility between teacher correction and student predictive support as token teachability . Fig 1 .A illustrates this distinction. Motivated by this diagnostic, we reformulate selective OPD as selecting learnable supervision rather than merely salient tokens. We instantiate this principle as Teachability-Aware OPD (TA-OPD), which scores each response position by support-aligned teacher–student disagreement and applies the OPD loss only to high-teachability positions. TA-OPD uses teacher and student token probabilities available during OPD, optionally approximated with a lightweight top- K support, and requires no reward model or verifier (Fig 1 .B). Across Qwen3 and Qwen2.5 teacher–student settings varying in scale, reasoning ability, and backbone, TA-OPD shows that token quality can outweigh token count. With only 5% retained tokens, TA-OPD often matches or surpasses full-token OPD and improves over budget-matched entropy- and divergence-based methods. These results support our central claim: OPD gains depend on teachable teacher signals, not merely dense supervision. Our contributions are summarized as follows. (i) we introduce a fixed-context diagnostic that measures same-context KL disagreement and links pre-update token signals to local OPD improvement. (ii) using this diagnostic, we formalize token teachability and show that raw KL disagreement conflates learnable and incompatible disagreement; only support-aligned disagreement reliably predicts useful learning. (iii) we propose Teachability-Aware OPD (TA-OPD), a lightweight token-position selection method that applies OPD loss to high-teachability positions without reward models or verifiers. (iv) we validate both the finding and the method across Qwen3 and Qwen2.5 settings, showing signficant performance with only 5% retained tokens. 2 Related Work LLM distillation. Knowledge distillation transfers a stronger teacher model’s behavior to a smaller student by matching outputs, intermediate representations, or generated trajectories ( 12 ; 38 ; 4 ) . For LLMs, distillation compresses reasoning, instruction-following, and alignment capabilities under limited compute ( 37 ; 24 ) . Off-policy distillation trains on teacher- or human-generated trajectories, which may diverge from student-visited states, while on-policy distillation supervises the student on its own rollouts with token-level teacher distributions ( 1 ; 8 ; 31 ) . We focus on this OPD setting to examine which teacher signals are truly learnable. Selective OPD. OPD provides dense token-level supervision, but recent work shows that useful signal is highly non-uniform ( 28 ) . Entropy-based criteria prioritize uncertain tokens ( 13 ; 10 ) , and divergence-based criteria highlight positions where the teacher strongly disagrees with the student ( 26 ; 20 ) . TIP ( 27 ) formalizes this with a two-axis taxonomy of student entropy and teacher–student divergence, enabling token-efficient training. Our work complements this line by assessing whether informative tokens are also learnable, showing that high divergence can mix support-aligned and off-support supervision. Teacher–student compatibility. Distillation quality depends on both teacher strength and compatibility with the student policy. Recent OPD analyses ( 14 ) suggest gains arise from alignment on high-probability student tokens and compatible reasoning patterns. Self-distillation ( 35 ; 1 ) and context-conditioned distillation ( 29 ; 32 ) further emphasize that effective supervision depends on information state and trajectory distribution. We operationalize compatibility at the token level by decomposing disagreement into learnable and incompatible components, reframing selective OPD as identifying signals that can