Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback

Ved Sriraman, Peihan Liu, Daniel Hsu, Adam Block · 2026

Read Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback by Ved Sriraman, Peihan Liu, Daniel Hsu, Adam Block (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30923.txt.

Introduction

Sriraman and three colleagues ask why on-policy distillation often beats behavior cloning when language models learn from imperfect teachers. In imitation learning, a student copies an expert's behavior; here the student only sees a noisy version of the expert, yet is judged against the clean expert's reward. That mismatch mirrors real training, where long chains of thought come from fallible demonstrators.

The authors prove a sharp separation. Learning offline from noisy demonstrations is fundamentally hard, with sample needs growing exponentially in the horizon. Learning online, through their new variant of on-policy distillation, needs only polynomially many samples. Experiments on synthetic and language tasks confirm the theory: the proposed method outperforms both offline cloning and existing distillation objectives under noisy feedback.

Catalog record

CREATORVed Sriraman, Peihan Liu, Daniel Hsu, Adam Block
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE222 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30923

Browse the full library · About the Tower of Babel