When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

Huaqing ZhangTsinghua University [email protected], Jingchu GaiCarnegie Mellon University [email protected] Juno KimUC Berkeley [email protected] Bingbin LiuHarvard University [email protected] Andrej RisteskiCarnegie Mellon University [email protected] · 2026

Read When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon by Huaqing ZhangTsinghua University [email protected], Jingchu GaiCarnegie Mellon University [email protected] Juno KimUC Berkeley [email protected] Bingbin LiuHarvard University [email protected] Andrej RisteskiCarnegie Mellon University [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30445.txt.

Introduction

Huaqing Zhang, Jingchu Gai, Juno Kim, Bingbin Liu, and Andrej Risteski ask when learning from an expert online beats learning offline for language models. Online imitation learning, especially on-policy distillation, often outperforms plain fine-tuning. The usual explanation blames error accumulation over long horizons. The authors challenge that story directly. The decisive factor is realizability, they argue. Realizability means the student can represent the expert's policy at all.

When the setting is realizable, offline learning already matches the expert. Experiments confirm offline methods reach expert performance. Non-realizable settings are where online interaction earns its keep. There, offline imitation hits an information-theoretic bottleneck. That bottleneck appears even at horizon one. A structural characterization of misspecification changes the picture. Relative to the reward, it lets online learning succeed anyway. High performance is provable despite large expert–student mismatch. The results reframe when to pay for online methods.

Catalog record

CREATORHuaqing ZhangTsinghua University [email protected], Jingchu GaiCarnegie Mellon University [email protected] Juno KimUC Berkeley [email protected] Bingbin LiuHarvard University [email protected] Andrej RisteskiCarnegie Mellon University [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30445

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon Huaqing ZhangTsinghua University [email protected], Jingchu GaiCarnegie Mellon University [email protected] Juno KimUC Berkeley [email protected] Bingbin LiuHarvard University [email protected] Andrej RisteskiCarnegie Mellon University [email protected] Abstract Online imitation learning (IL), particularly on-policy distillation, has emerged as a strong LLM post-training approach, often outperforming offline supervised fine-tuning (SFT). Yet a principled understanding of when and why online interaction helps remains unclear. In this work, we challenge the view that error accumulation is the main source of online IL’s advantage, and instead show that the benefits of online interaction depend critically on whether the setting is realizable, i.e., whether the student policy class can represent the expert policy. Under realizability, we empirically find that offline IL already matches expert performance. In contrast, in non-realizable (misspecified) settings, we prove that offline IL encounters an information-theoretic bottleneck even when horizon H = 1 H=1 , and propose a structural characterization of misspecification relative to the reward, under which online IL provably achieves high performance despite a large distributional mismatch between the expert and student policies. * * footnotetext: Equal contribution. 1 Introduction Imitation learning (IL) trains a student policy from expert demonstrations, enabling the student to achieve strong performance without access to explicit reward signals. For post-training of large language models (LLMs), the canonical offline imitation learning method is supervised fine-tuning (SFT), where the learner imitates an expert policy using a fixed dataset of expert demonstrations, such as human-written responses or traces generated by a stronger model. However, such offline approaches are reported to suffer from limitations such as poor generalization [ 7 , 21 , 30 ] and severe catastrophic forgetting [ 5 ] . Online imitation learning offers a promising alternative for mitigating these issues in post-training. In the online IL setting, the learner actively interacts with the environment and queries an expert to obtain feedback [ 48 ] . A prominent online IL approach for LLM post-training is on-policy distillation , where the student samples from its current policy and minimizes the reverse-KL divergence to the expert distribution [ 29 , 2 , 20 ] . Although recent studies report the empirical superiority of online IL in LLM post-training [ 63 , 35 ] , a principled understanding of when and why this advantage arises remains limited. A classical explanation for the advantage of online IL is its ability to mitigate error accumulation: offline IL can suffer from errors that compound quadratically over the horizon, whereas DAgger-style online algorithms improve this dependence under a recoverability assumption [ 46 , 48 ] . However, recent theory by 13 shows that under the standard log-likelihood objective used in SFT, offline IL can match the sample-complexity guarantees of DAgger in realizable settings, where the expert policy is contained in the student policy class. This result calls into question whether error accumulation alone can fully explain the practical advantage of online IL in LLM training. In this work, we identify non-realizability (or misspecification ) as an orthogonal source of online IL’s advantage: the student policy class is too restricted, e.g., due to a much smaller model size, to represent the expert policy. Unlike the classical error-accumulation view, which concerns long-horizon compounding, this advantage can already arise in contextual bandits with horizon H=1 . We first study the realizable setting as a sanity check, and empirically show that offline IL already suffices in this setting ( Section 4 ). Specifically, we consider both synthetic (Countdown [ 39 ] ) and real-world math reasoning (GSM8K [ 8 ] and DeepScaleR [ 36 ] ) tasks. For each task we first apply reinforcement learning (RL) to a base model to obtain the expert, and then compare SFT and on-policy distillation initialized from the same base model. By construction, this setup guarantees realizability, since the expert policy is itself obtained within the same model family. Our results show that the SFT-trained student matches the expert’s performance, and on-policy distillation offers no further gains in either accuracy or training speed. This finding is consistent with the theory of 13 , which predicts no statistical-complexity gap between offline and online IL in realizable settings, and further indicates that other potential sources of online IL advantage, such as optimization effects or problem-specific structures, do not yield an empirical advantage in this regime either. Taken together, these results further motivate our focus on the non-realizable setting as the key regime for understanding the benefits of online imitation learning. In non-realizable settings, we develop an account of online IL’s advantage that is orthogonal to the classical error-accumulation argument. We focus on a simplified yet informative contextual-bandit framework with horizon H=1 , where long-horizon compounding is absent by construction, yet online IL still gains an advantage from misspecification. We first show, both theoretically and empirically, that prior analyses that control the performance gap via distributional discrepancies between the expert and student policies [ 13 , 45 , 43 ] can be overly pessimistic under misspecification, and reward-dependent analyses are necessary. For offline IL, we identify an information-theoretic limitation specific to non-realizable settings: the expert policy may place too little probability mass on responses that the restricted student class can learn. For online IL, we further introduce a structural condition that explicitly captures the misspecification structure relative to the reward ( 1 ), under which online IL can provably achieve small performance suboptimality despite large distributional discrepancy. At a high level, the condition states that the expert’s signal should either be aligned with the actual reward, or deviate from it only in ways that the restricted student class cannot realize. To summarize, our main contribution is to identify non-realizability as a key factor underlying the advantage of online over offline imitation learning in language model post-training. Specifically, 1. In realizable settings, we empirically verify that offline IL (SFT) fully recovers expert performance, and that online IL (on-policy distillation) yields neither accuracy gains nor faster learning ( Section 4 ). 2. In non-realizable settings, we show that prior discrepancy-based analyses can be overly pessimistic ( Section 5.1 ), and reward-dependent analyses are necessary to explain the success of IL. 3. By analyzing the reward-dependent structure of misspecification, we identify an information-theoretic limitation of offline IL in this regime ( Section 5.2 ) and propose a structural condition characterizing when online IL can remain effective despite misspecification ( Section 5.3 ). 2 Preliminaries For a set \mathcal{X} , let \Delta(\mathcal{X}) denote the set of all probability distributions over \mathcal{X} . For two distributions P,Q over \mathcal{X} , their Kullback–Leibler divergence is defined by D_{\mathrm{KL}}\left({P}\|{Q}\right)\coloneqq\mathbb{E}_{x\sim P}\!\left[\log\frac{P(x)}{Q(x)}\right] , and their total variation distance is defined by D_{\text{TV}}(P,Q)\coloneqq\sup_{A\subseteq\mathcal{X}}|P(A)-Q(A)|=\frac{1}{2}\sum_{x\in\mathcal{X}}|P(x)-Q(x)| . We use standard asymptotic notation throughout, and we write a\lesssim b as shorthand for a=O(b) , and a\gtrsim b as shorthand for a=\Omega(b) . 2.1 Problem Setting To provide a clean theoretical sandbox, we abstract language model post-training as a contextual bandit problem [ 41 , 62 , 14 ] . Let \mathcal{X} represent the set of all possible prompts (contexts) and \mathcal{Y} the set of generated responses (actions). A policy \pi:\mathcal{X}\to\Delta(\mathcal{Y}) specifies a conditional distribution \pi(y\mid x) over responses y given a prompt x . Let r(x,y):\mathcal{X}\times\mathcal{Y}\to[0,1] be a

Read the complete text