Fast and Effective On-policy Distillation from Reasoning Prefixes
Dongxu ZhangZhichao YangSepehr JanghorbaniJun Han Andrew Ressler II Qian Qian Gregory D. Lyng, Sanjit Singh Batra Robert E. Tillman Optum AI · 2026
Read Fast and Effective On-policy Distillation from Reasoning Prefixes by Dongxu ZhangZhichao YangSepehr JanghorbaniJun Han Andrew Ressler II Qian Qian Gregory D. Lyng, Sanjit Singh Batra Robert E. Tillman Optum AI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15260.txt.
Description
Zhang and colleagues propose on-policy prefix distillation, a fast tweak of on-policy distillation. On-policy distillation draws paths from the student model and has a teacher check them token by token. It avoids reliance solely on proven final rewards, and can transfer better than off-policy distillation. Costly live sampling of the student policy while training sharply raises training cost, above all for long answers.
An early look shows training signals often cluster in the prefix of each output. Even a short teacher-generated prefix can greatly help the student produce the correct answer. Building on these findings, the team applies the training objective only to prefixes of student outputs and stops each sampling early. Tests on AI-for-Math and out-of-domain test sets show it matches full on-policy distillation while cutting training FLOP 2 to 47 times.
Catalog record
| CREATOR | Dongxu ZhangZhichao YangSepehr JanghorbaniJun Han Andrew Ressler II Qian Qian Gregory D. Lyng, Sanjit Singh Batra Robert E. Tillman Optum AI |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-15260