Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level

Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai · 2026

Read Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level by Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06387.txt.

Description

Nan Jia and eight co-authors study on-policy distillation, a method that trains a student model on its own output with token-level feedback from a teacher. They find three weaknesses in the standard approach: noisy updates, vanishing signals where the advantage is zero, and weak exploration when corrective feedback is thin.

Their fix, Asymmetric On-Policy Distillation, replaces ineffective negative reinforcement with a local divergence penalty in non-positive regions while keeping positive reinforcement. On math reasoning benchmarks it beats standard distillation, keeps higher entropy during training, and retains skills better during later tool-use adaptation.

Catalog record

CREATORNan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06387

Browse the full library · About the Tower of Babel