A Note on Hybrid Online Reinforcement and Imitation Learning for LLMs: Formulations and Algorithms
Yingru Li, Ziniu Li, Jiacai Liu · 2025
Read A Note on Hybrid Online Reinforcement and Imitation Learning for LLMs: Formulations and Algorithms by Yingru Li, Ziniu Li, Jiacai Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23097.txt.
Introduction
Yingru Li, Ziniu Li, and Jiacai Liu write a note unifying two standard ways of fine-tuning large language models. One is imitation learning, which copies demonstrated behavior. The other is reinforcement learning, which chases task rewards. Rather than treating the two as separate pipelines, they write one composite objective that combines a trajectory-level divergence penalty against demonstrations with the reward signal, then examine its gradient.
The gradient splits naturally into two pieces. One is a dense gradient for token-level imitation, which turns out to have a closed-form expression at the level of output logits, so it can be implemented efficiently on graphics hardware. The other is a sparse gradient for long-horizon reward optimization, which must be estimated by Monte Carlo sampling. This decomposition gives designers one framework where the supervised part is computed exactly while the exploratory part is sampled.
Catalog record
| CREATOR | Yingru Li, Ziniu Li, Jiacai Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-23097
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Note on Hybrid Online Reinforcement and Imitation Learning for LLMs: Formulations and Algorithms Thanks: First version: December 15, 2025 Yingru Li, Ziniu Li, Jiacai Liu Abstract We present a unified framework for Large Language Model (LLM) fine-tuning that integrates Imitation Learning and Reinforcement Learning . By analyzing the gradient of a composite objective combining trajectory-level KL divergence with task rewards, we derive a natural decomposition into two components: (1) an analytically computable Dense Gradient for token-level imitation, and (2) a Monte Carlo estimated Sparse Gradient for long-horizon reward optimization. The Dense Gradient admits a closed-form logit-level formula, enabling efficient GPU implementation. 1 Introduction Knowledge Distillation (KD) [ 1 ] and Reinforcement Learning (RL) are two fundamental approaches to training Large Language Models. Standard KD minimizes divergence on static offline data via teacher forcing, leading to train-inference distribution mismatch (also known as exposure bias or covariate shift) [ 2 ] : during training, the student conditions on ground-truth history, while during inference it conditions on its own generations. This mismatch compounds over long sequences, causing errors to accumulate—a phenomenon well-studied in online imitation learning [ 3 , 4 ] . The DAgger algorithm [ 3 ] addresses distribution mismatch by iteratively collecting data under the learner’s own policy while querying the expert for labels. Recent theoretical work [ 4 ] establishes that interactive access to an expert provides provable statistical advantages over passive behavior cloning: under \mu -recoverability assumptions, interactive learners achieve suboptimality \widetilde{O}(\mu|S|H/N) , while non-interactive learners suffer \Omega(|S|H^{2}/N) —a quadratic-to-linear improvement in horizon dependence. RL addresses the distribution mismatch by training on self-generated data, but introduces new challenges: high variance from sparse, trajectory-level rewards and risks of reward hacking. Recent work on on-policy distillation [ 5 , 6 ] attempts to bridge these approaches. In this work, we analyze the gradient structure of a unified objective that combines on-policy KL minimization with reward maximization, revealing a natural decomposition that enables efficient implementation. Contributions. 1. We derive the exact gradient of trajectory-level KL + reward objectives, proving it decomposes into Dense (analytic) and Sparse (sampled) terms (Theorem 1 ). 2. We provide an efficient logit-level gradient formula amenable to GPU implementation (Proposition 5 ). 3. We establish mathematical equivalence to KL-regularized RLHF while clarifying interpretational differences (Section 5.2 ). 4. We discuss training curriculum implications for the reward weight \lambda (Section 5.1 ). 2 Problem Formulation 2.1 Setup and Notation Definition 1 (Autoregressive Policy) . An LLM policy \pi generates response y=(y_{1},\ldots,y_{T}) given prompt x with probability: \pi(y|x)=\prod_{t=1}^{T}\pi(y_{t}|x,y_{<t}) (1) where y_{<t}=(y_{1},\ldots,y_{t-1}) and each \pi(\cdot|x,y_{<t}) is a distribution over vocabulary \mathcal{V} . Definition 2 (Instantaneous Divergence Cost) . At step t , the log-ratio between student \pi_{\theta} and reference \pi_{\text{ref}} is: c_{t}=\log\pi_{\theta}(y_{t}|x,y_{<t})-\log\pi_{\text{ref}}(y_{t}|x,y_{<t}) (2) 2.2 The Hybrid Objective We minimize a cost combining distribution matching with reward maximization: \mathcal{J}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[D_{\text{KL}}\big(\pi_{\theta}(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\big)-\lambda\,\mathbb{E}_{y\sim\pi_{\theta}}[r(x,y)]\right] (3) where \lambda\geq 0 controls the trade-off and r(x,y) is a black-box reward. Using the autoregressive factorization, the trajectory KL decomposes as: D_{\text{KL}}(\pi_{\theta}(\cdot|x)\|\pi_{\text{ref}}(\cdot|x))=\mathbb{E}_{y\sim\pi_{\theta}}\left[\sum_{t=1}^{T}c_{t}\right] (4) Thus the objective becomes: \mathcal{J}(\theta)=\mathbb{E}_{x,y\sim\pi_{\theta}}\left[\sum_{t=1}^{T}c_{t}-\lambda r(x,y)\right] (5) Remark 1 (Connection to Online Imitation Learning) . Our framework can be viewed as a continuous relaxation of DAgger [ 3 ] . While DAgger iteratively collects data under the learner’s distribution and trains via supervised learning, we directly optimize a trajectory-level objective under the learner’s distribution. The Dense Term provides the imitation signal (analogous to DAgger’s supervised loss), while on-policy sampling addresses the distribution mismatch. 3 Main Results We now state our main theoretical results. All proofs are deferred to Appendix A . Definition 3 (Future Return) . The future return from step t+1 is: G_{t+1}=\sum_{k=t+1}^{T}c_{k}-\lambda r(x,y) (6) This captures future divergence costs plus the (negative) terminal reward. Theorem 1 (Gradient Decomposition) . The gradient of objective ( 3 ) at step t decomposes as: \boxed{\nabla_{\theta}\mathcal{J}_{t}(\theta)=\underbrace{\mathbb{E}\left[\nabla_{\theta}\log\pi_{\theta}(y_{t}|x,y_{<t})\cdot c_{t}\right]}_{\textbf{Dense Term}}+\underbrace{\mathbb{E}\left[\nabla_{\theta}\log\pi_{\theta}(y_{t}|x,y_{<t})\cdot G_{t+1}\right]}_{\textbf{Sparse Term}}} (7) where the expectation is over x\sim\mathcal{D} and y\sim\pi_{\theta}(\cdot|x) . The proof relies on two key lemmas: Lemma 2 (Vanishing Score Function) . For any context (x,y_{<t}) : \mathbb{E}_{y_{t}\sim\pi_{\theta}(\cdot|x,y_{<t})}\left[\nabla_{\theta}\log\pi_{\theta}(y_{t}|x,y_{<t})\right]=0 (8) Lemma 3 (Causality) . For k<t , past costs do not contribute to the gradient at step t : \mathbb{E}_{y}\left[c_{k}\cdot\nabla_{\theta}\log\pi_{\theta}(y_{t}|x,y_{<t})\right]=0 (9) The Dense Term has a remarkable closed-form property: Proposition 4 (Dense Term Equals Token-Level KL Gradient) . \mathbb{E}_{y_{t}\sim\pi_{\theta}}\left[\nabla_{\theta}\log\pi_{\theta}(y_{t}|x,y_{<t})\cdot c_{t}\right]=\nabla_{\theta}D_{\text{KL}}\big(\pi_{\theta}(\cdot|x,y_{<t})\,\|\,\pi_{\text{ref}}(\cdot|x,y_{<t})\big) (10) This means the Dense Term can be computed analytically by summing over the finite vocabulary \mathcal{V} , without Monte Carlo sampling. Proposition 5 (Logit-Level Gradient) . Let p=\mathrm{softmax}(z) be the student distribution and q the teacher distribution. The gradient of KL with respect to logits z is: \boxed{\nabla_{z}D_{\text{KL}}(p\|q)=p\odot\left(\log p-\log q-D_{\text{KL}}(p\|q)\cdot\mathbf{1}\right)} (11) where \odot denotes element-wise multiplication. This formula enables efficient GPU implementation for both full-vocabulary and Top- K computation. 3.1 Interpretation of Terms Term Dense (Imitation) Sparse (RL) Signal source Current token distribution Future trajectory + reward Computation Analytic (sum over \mathcal{V} ) Monte Carlo sampling Variance None (deterministic given context) Scales with horizon T Contains reward? No Yes 3.2 Discounted Future Return To control the variance-bias trade-off, we introduce discount factor \gamma\in[0,1] : G_{t+1}^{(\gamma)}=\sum_{k=t+1}^{T}\gamma^{k-t}c_{k}-\lambda r(x,y) (12) Special cases: • \gamma=0 : G_{t+1}^{(0)}=-\lambda r(x,y) (only terminal reward, no future KL) • \gamma=1 : G_{t+1}^{(1)}=\sum_{k>t}c_{k}-\lambda r (full trajectory) Remark 2 (Discounting Convention) . The exponent \gamma^{k-t} (rather than \gamma^{k-t-1} ) means the immediate next step c_{t+1} is discounted by \gamma relative to the current cost c_{t} . This provides a natural interpolation: \gamma=0 isolates each step from all future KL costs, while \gamma=1 gives full trajectory credit assignment. 4 Algorithm Algorithm 1 Hybrid Online Reinforcement & Imitation Learning 1: Input: Student \pi_{\theta} , Teacher \pi_{\text{ref}} , Reward r , Group size K , Discount \gamma , Weight \lambda 2: for each training iteration do 3: Sample prompt x\sim\mathcal{D} 4: Generate K responses \{y^{(i)}\}_{i=1}^{K} from \pi_{\theta}(\cdot|x) // Group rollout 5: Initialize gradient accumulator g\leftarrow 0 6: for each response i , step t do 7: c_{t}^{(i)}\leftarrow\log\pi_{\theta}(y_{t}^{(i)}|x,y_{<t}^{(i)})-\log\pi_{\text{ref}}(y_{t}^{(i)}|x,y_{<t}^{(i)}) 8: end for 9: Compute rewards: R^{(i)}\leftarrow r(x,y^{(i)}) 10: for each response i , step t do 11: G_{t+1}^{(i)}\leftarrow\sum_{k=t+1}^{T^{(i)}}\gamma^{k-t}c_{k}^{(i)}-\lambda R^{(i)} 12: end for 13: for each response i , step t do 14: Dense: g_{\text{dense}}\leftarrow p\odot(\log p-\log q-D_{\text{KL}}(p\|q)\cdot\mathbf{1}) // Eq. ( 11 ) 15: Sparse: g_{\text{sparse}}\leftarrow\text{PolicyGradient}(\pi_{\theta},y_{t}^{(i)},G_{t+1}^{(i)}) 16: g\leftarrow g+g_{\text{dense}}+g_{\text{sparse}} // Sum over all (i, t) 17: end for 18: \theta\leftarrow\theta-\eta\cdot g/K // Average over group 19: end for Remark 3 (Compatibility with Group Rollout Infrastructure) . The algorithm is fully compatible with standard group rollout RL infrastructure. The group size K corresponds to the number of responses sampled per prompt in existing implementations (e.g., GRPO, RLOO). The only additions are: (1) computing token-level log-ratios c_{t} against the teacher, and (2) accumulating the Dense gradient alongside the standard policy gradient. Both operations integrate naturally into existing training pipelines. 5 Discussion 5.1 Training Curriculum for \lambda The dual interpretation of our framework suggests a natural training curriculum: • Early training ( \lambda small, equivalently \beta large): Prioritize imitation. Dense Term dominates, providing stable gradients toward teacher behavior (“cold start”). • Late training ( \lambda large, equivalently \beta small): Prioritize reward. Sparse Term becomes influential, enabling discovery of reward-maximizing behaviors beyond the teacher. A simple schedule: \lambda(t)=\lambda_{0}(1+\alpha t) where t is training step. In the limit \lambda\to\infty , the framework approaches pure RL, but the Dense Term provides ongoing regularization toward the expert. 5.2 Relationship to KL-Regularized RLHF