Fast and Effective On-policy Distillation from Reasoning Prefixes

Dongxu ZhangZhichao YangSepehr JanghorbaniJun Han Andrew Ressler II Qian Qian Gregory D. Lyng, Sanjit Singh Batra Robert E. Tillman Optum AI · 2026

Read Fast and Effective On-policy Distillation from Reasoning Prefixes by Dongxu ZhangZhichao YangSepehr JanghorbaniJun Han Andrew Ressler II Qian Qian Gregory D. Lyng, Sanjit Singh Batra Robert E. Tillman Optum AI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15260.txt.

Description

Zhang and colleagues propose on-policy prefix distillation, a fast tweak of on-policy distillation. On-policy distillation draws paths from the student model and has a teacher check them token by token. It avoids reliance solely on proven final rewards, and can transfer better than off-policy distillation. Costly live sampling of the student policy while training sharply raises training cost, above all for long answers.

An early look shows training signals often cluster in the prefix of each output. Even a short teacher-generated prefix can greatly help the student produce the correct answer. Building on these findings, the team applies the training objective only to prefixes of student outputs and stops each sampling early. Tests on AI-for-Math and out-of-domain test sets show it matches full on-policy distillation while cutting training FLOP 2 to 47 times.

Catalog record

CREATORDongxu ZhangZhichao YangSepehr JanghorbaniJun Han Andrew Ressler II Qian Qian Gregory D. Lyng, Sanjit Singh Batra Robert E. Tillman Optum AI
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15260

Browse the full library · About the Tower of Babel