The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation

Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Qinyuan Ye, Chien-Sheng Wu · 2026

Read The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation by Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Qinyuan Ye, Chien-Sheng Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16830.txt.

Introduction

Zhang and colleagues show on-policy distillation lifts task accuracy while trapping models in severe overconfidence. Teacher supervision uses hidden training context, but live models must report confidence from live data alone. Teacher-led success is therefore not a valid trust target, and hidden context induces entropy collapse and bias. CaOPD gauges seen trust from rollouts, swaps self-told trust with that student-grounded target, and distills the revised reply. Across models and domains, CaOPD reaches Pareto-optimal calibration with strong capability, generalizing under shift and ongoing learning.

Catalog record

CREATORJiaxin Zhang, Xiangyu Peng, Qinglin Chen, Qinyuan Ye, Chien-Sheng Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE121169
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16830

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Qinyuan Ye, Chien-Sheng Wu Abstract On-policy distillation (OPD) is an increasingly important paradigm for post-training language models. However, we identify a pervasive Scaling Law of Miscalibration : while OPD effectively improves task accuracy, it systematically traps models in severe overconfidence. We trace this failure to an information mismatch: teacher supervision is formed under privileged context available during training, whereas the deployed model must report confidence using only deployment-time information. We formalize this perspective theoretically, showing that teacher-conditioned success is generally not a valid target for deployment-time confidence and that helpful privileged context induces entropy collapse and a systematic optimism bias . To address this, we propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts, replaces self-reported confidence with this student-grounded target, and distills the revised response through the same self-distillation pipeline. Experiments across various models and domains show that CaOPD achieves Pareto-optimal calibration while maintaining competitive capability, generalizing robustly under out-of-distribution and continual learning. Our findings highlight that capability distillation does not imply calibrated confidence, and that confidence should be treated as an essential objective in post-training. “… AI tools do not rate their own confidence accurately. And this lowers their usefulness. We would appreciate more honest AIs.” — Terence Tao, The Atlantic , “The Edge of Mathematics” (2026) Figure 1: The Scaling Law of Miscalibration. (Left) Mean Confidence vs. Accuracy on Science Q&A. Almost all modern LLMs are trapped in the red Overconfidence Zone , exhibiting massive calibration gaps. Scaling up capability does not resolve this blind optimism. (Right) CaOPD structurally eliminates this bias by decoupling capability from calibration. It pulls the model back to the ideal calibration line, enabling a compact 8B model to achieve top-tier reliability (1-Brier Score) that rivals frontier LLMs (APIs & Open-weight Models). 1 Introduction As LLMs achieve breakthroughs in complex reasoning and problem-solving, their role is shifting from text generators to autonomous agents and scientific co-pilots. For AI to be a trustworthy collaborator in rigorous domains, it must possess reliable uncertainty signaling , the ability to accurately self-assess and express doubt. Unfortunately, modern LLMs consistently fail this criterion. As shown in Figure 1 , we observe a pervasive Scaling Law of Miscalibration : even the most advanced proprietary models (e.g., GPT ( 1 ) , Claude ( 4 ) , Gemini ( 44 ) ) and scaled-up open-weight models (e.g., DeepSeek ( 30 ) , Kimi ( 45 ) ) exhibit severe overconfidence. Scaling up capability does not resolve this blind optimism. This widespread pathology is deeply rooted in the modern post-training paradigm. To achieve advanced capabilities, developers increasingly rely on On-Policy Distillation (OPD) ( 2 ; 31 ; 52 ) and self-distillation frameworks ( 36 ; 21 ; 54 ) . Methods such as SDPO ( 17 ) and SDFT ( 39 ) generate high-quality training trajectories by conditioning a teacher policy on privileged contexts , such as environment verifier feedback, expert demonstrations, or ground-truth solutions. While OPD is phenomenally successful at transferring capability, it completely destabilizes confidence calibration. Indeed, as recently argued by 20 , modern language models are effectively trapped in a "test-taking mode," incentivized to output plausible falsehoods rather than admitting uncertainty. We trace the algorithmic root of this epidemic directly to the OPD paradigm itself. We demonstrate that this overconfidence is not a random artifact but a mathematical consequence of OPD, rooted in a fundamental information asymmetry between training and deployment ( 35 ) . The teacher operates “open-book,” conditioning on privileged context to produce low-entropy reasoning paths; the deployed model operates “closed-book” with only the user prompt. Minimizing the divergence against this privileged distribution forces the student to artificially sharpen its logits ( Entropy Collapse ; 33 ) and to inherit the declarative style of success-filtered trajectories ( Optimism Bias ; 3 ). In short: capability transfers by imitation, but confidence cannot safely transfer across asymmetric information states. Existing efforts to tame overconfidence typically rely on Reinforcement Learning (RL) with explicit reward shaping, such as penalizing Brier scores via RLCR ( 10 ) . While these methods can suppress absolute confidence, they suffer from a severe capability tax ( 5 ) , the models become overly conservative and sacrifice fundamental reasoning accuracy to avoid RL penalties. To break this capability-calibration trade-off, we propose Calibration-Aware On-Policy Distillation (CaOPD) . Rather than fighting the optimizer with complex reward shaping, CaOPD addresses the root cause via target decoupling . We separate what the model should answer from how certain it should be. We sample multiple rollouts from the student policy, which under SDPO are already generated at zero extra cost, to compute a student-grounded, unbiased empirical success rate. We then perform a non-invasive target replacement : we preserve the teacher’s high-quality trajectory for capability cloning, but overwrite the implicit confidence token with the student’s actual confidence. By distilling this revised joint target, CaOPD effectively amortizes the heavy computational expense of test-time confidence estimation directly into the parameters of a single-pass inference model. Our main contributions are summarized as follows: • Identifying the Scaling Law of Miscalibration: We empirically demonstrate that standard capability scaling and OPD inherently induce systematic overconfidence, a pervasive pathology that traps frontier LLMs in blind optimism. • Theoretical Formalization: We mathematically formalize the information asymmetry in standard OPD, proving that distilling from a privileged teacher context inherently induces entropy collapse and optimism bias . • The CaOPD Framework: We propose CaOPD, a compute-efficient post-training framework that decouples behavior imitation from confidence supervision. CaOPD amortizes expensive test-time empirical confidence into a single-pass model. • Pareto-Optimal Performance and Robustness: Across OPD regimes (SDFT, SDPO) and domains (Science Q&A, Tool Use), CaOPD achieves Pareto-optimal calibration without the capability tax of RL-based methods, generalizing robustly under OOD shifts and preventing "calibration forgetting" during continual learning. 2 Problem Formulation and Preliminaries 2.1 On-Policy Distillation and Self-Distillation Let \mathcal{X} denote the space of input queries. To study confidence calibration explicitly, we augment the standard generation format with a verbalized confidence instruction ( 19 ; 55 ; 46 ) , prompting the model to append an explicit confidence statement after its reasoning. Each complete generation y=(y_{1},\dots,y_{T}) is partitioned into two contiguous segments: y=(a,c) , where a=(y_{1},\dots,y_{T_{a}}) is the reasoning trajectory (including the final answer), and c=(y_{T_{a}+1},\dots,y_{T}) is the confidence segment , the token subsequence encoding a verbalized confidence statement such as “ Confidence: 0.85 ”. We write \mathrm{val}(c)\in[0,1] for the scalar confidence score extracted by parsing the confidence segment c . Modern OPD methods employ a self-distillation architecture in which a single model plays both roles: Student policy \pi_{\theta}(\cdot\mid x) : generates y conditioned only on the input query x , simulating deployment conditions; Teacher policy \pi_{\theta}(\cdot\mid x,z) : the same model, but additionally conditioned on a privileged context z\sim\mathcal{Z}(x) . Depending on the OPD paradigm, z may instantiate as an expert demonstration, environment feedback, or a ground-truth solution. The shared structural property is that the teacher is conditioned on strictly richer evidence than the deployed model will ever observe. 2.2 The Ideal Calibration Objective Let R:\mathcal{X}\times\mathcal{A}\to\{0,1\} be an objective verification function, where R(x,a)=1 indicates that the reasoning trajectory a yields a correct

Read the complete text