Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai · 2026
Read Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level by Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06387.txt.
Description
Nan Jia and eight co-authors study on-policy distillation, a method that trains a student model on its own output with token-level feedback from a teacher. They find three weaknesses in the standard approach: noisy updates, vanishing signals where the advantage is zero, and weak exploration when corrective feedback is thin.
Their fix, Asymmetric On-Policy Distillation, replaces ineffective negative reinforcement with a local divergence penalty in non-positive regions while keeping positive reinforcement. On math reasoning benchmarks it beats standard distillation, keeps higher entropy during training, and retains skills better during later tool-use adaptation.
Catalog record
| CREATOR | Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06387