Learning from Language Feedback via Variational Policy Distillation
Yang Li, Erik Nijkamp, Semih Yavuz, Shafiq Joty · 2026
Read Learning from Language Feedback via Variational Policy Distillation by Yang Li, Erik Nijkamp, Semih Yavuz, Shafiq Joty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15113.txt.
Introduction
Yang Li and three others propose Variational Policy Distillation, a way to learn from language feedback on reasoning tasks. Reinforcement learning from verifiable rewards gives sparse outcome signals, and self-distillation teachers plateau as students improve. VPD treats feedback learning as variational expectation-maximization. The teacher refines itself on trajectory outcomes through a trust-region update. The student absorbs the dense token guidance on its own rollouts. Tests on science problems and code generation show steady gains.
Catalog record
| CREATOR | Yang Li, Erik Nijkamp, Semih Yavuz, Shafiq Joty |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85312 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning from Language Feedback via Variational Policy Distillation Yang Li, Erik Nijkamp, Semih Yavuz, Shafiq Joty Abstract Reinforcement learning from verifiable rewards (RLVR) suffers from sparse outcome signals, creating severe exploration bottlenecks on complex reasoning tasks. Recent on-policy self-distillation methods attempt to address this by utilizing language feedback to generate dense, token-level supervision. However, these approaches rely on a fixed, passive teacher to interpret the feedback. As the student policy improves, the teacher’s zero-shot assessment capabilities plateau, ultimately halting further learning. To overcome this, we propose Variational Policy Distillation (VPD), a framework that formalizes learning from language feedback as a Variational Expectation-Maximization (EM) problem. VPD co-evolves both policies: in the E-step, the teacher is actively refined on trajectory outcomes via an adaptive trust-region update, translating textual feedback into a dynamically improved target token distribution. In the M-step, the student internalizes this dense distributional guidance on its own on-policy rollouts. By continuously improving the teacher’s ability to extract actionable signals from textual critique, VPD overcomes the limitations of passive distillation. Evaluated across diverse sources of diagnostic feedback on scientific reasoning and code generation tasks, VPD consistently outperforms both standard RLVR and existing self-distillation baselines. Finally, by stress-testing our framework on rigid mathematical reasoning and cold-start regimes, we illuminate the fundamental bounds of feedback-driven self-distillation compared to pure environment-driven RL. 1 Introduction Recent leaps in the reasoning capabilities of large language models (LLMs) have been largely driven by reinforcement learning from verifiable rewards (RLVR) ( 9 ; 33 ; 38 ) . By optimizing models against objective, outcome-based correctness, RLVR avoids the high cost of human preference data in standard RLHF ( 22 ) . However, standard policy gradient methods like GRPO ( 33 ) and their variants ( 50 ; 43 ) rely almost entirely on sparse, binary outcome signals. This creates a severe credit assignment bottleneck: a minor arithmetic mistake in a complex derivation receives the same zero-reward as a completely nonsensical hallucination. Consequently, standard outcome-based RL is notoriously sample inefficient ( 48 ; 51 ) . On hard problems where the model’s initial success rate is near zero, on-policy algorithms face an extreme exploration bottleneck: they receive zero positive learning signal regardless of how many rollouts are sampled, entirely wasting the valuable latent information embedded in near-miss trajectories ( 26 ; 32 ) . To overcome this sparsity, a promising paradigm is to learn directly from language feedback . In many real-world and agentic settings, failure is accompanied by rich textual diagnostics, such as automated critiques from a stronger LLM, compiler error traces, or user corrections. This textual feedback can potentially provide exactly the dense, localized supervision that scalar rewards lack, pointing out not just that an attempt failed, but why and how it should be fixed. Leveraging this rich feedback effectively, however, remains an open challenge. Off-policy methods, such as supervised fine-tuning on expert traces or feedback-revised trajectories, suffer from distribution mismatch: the student model often lacks the internal capacity to faithfully reproduce the external teacher’s reasoning, leading to copycat behavior without genuine comprehension ( 17 ; 29 ) . Recently, on-policy self-distillation methods like SDPO ( 10 ) and OPSD ( 49 ) condition the model itself on language feedback to act as a “self-teacher,” distilling feedback-informed next-token predictions back into the unconditioned policy. Crucially, by sampling directly from the student’s own distribution, these methods avoid the severe train-inference distribution mismatch of off-policy approaches ( 34 ; 1 ) , allowing the teacher to act as an internal critic providing dense, token-level learning signals ( 10 ) . Yet, existing self-distillation approaches suffer from a critical flaw: they treat the feedback-conditioned self-teacher as a fixed, passive function . The quality of the distillation signal depends entirely on the model’s zero-shot ability to parse and exploit the language feedback. If the critique is noisy, or if the model cannot yet map natural language hints to structural token adjustments, the self-teacher’s guidance can become counterproductive. Furthermore, as the student internalizes basic corrections, the zero-shot advantage of appending feedback diminishes. Since a passive teacher is never explicitly trained to be a sharper critic, its ability to distinguish between increasingly subtle reasoning errors plateaus, ultimately starving the student of further meaningful gradients. To address this, we propose Variational Policy Distillation (VPD) , a principled framework that frames learning from language feedback as a variational inference problem. Instead of taking the self-teacher’s feedback interpretation for granted, VPD treats the feedback-conditioned model as an approximate posterior over correct solutions that must be actively optimized alongside the student policy. This variational perspective naturally yields an Expectation-Maximization (EM) algorithm that enables the teacher and student to co-evolve: • E-step (Teacher Refinement): We actively train the teacher’s ability to interpret language feedback. By optimizing the teacher to distinguish between successful and failed trajectories given the rich textual critique, we effectively teach the teacher how to read and leverage the feedback. • M-step (Student Optimization): We distill this refined knowledge back into the student. By minimizing the token-level KL divergence against the improved teacher on its own on-policy rollouts, the student internalizes this dense learning signal to succeed zero-shot at deployment. By ensuring the teacher’s assessment capabilities scale alongside the student’s reasoning, VPD extracts significantly more value from language feedback than passive distillation methods. We summarize our contributions as follows: 1. We formalize on-policy learning from language feedback as a Variational EM procedure. This introduces an explicit, feedback-aware teacher update (absent from prior self-distillation methods) implemented via unpaired preference optimization. By dynamically anchoring the teacher to the current student, we enforce an adaptive trust region that ensures highly stable on-policy KL distillation over a shared-weight network. 2. We present a comprehensive empirical study instantiating VPD across three sources of diagnostic feedback: deterministic environment verifiers, contrastive sibling rollouts, and autonomous self-critique. Evaluated on benchmarks spanning competitive programming and scientific reasoning, VPD consistently outperforms standard RLVR and self-distillation baselines. 3. We characterize the fundamental regimes in which language-feedback distillation outperforms sparse RL, and where it does not. By stress-testing our framework on base-model cold-start scenarios and challenging mathematical reasoning, we empirically establish the bounds of language-driven self-distillation. While VPD significantly mitigates and delays the training collapse typically observed in these settings, our results demonstrate that pure sparse RL ultimately remains the most effective paradigm. 2 Preliminaries Reinforcement Learning from Verifiable Rewards. We model language generation as a contextual bandit problem where the context is the user prompt x\in\mathcal{X} . A language model, parameterized by \theta , represents a policy \pi_{\theta} that generates a response y=(y_{1},\ldots,y_{T}) autoregressively: y\sim\pi_{\theta}(\cdot\mid x) . In the RLVR framework, an environment or rule-based verifier assesses the final response and assigns a scalar outcome reward r(x,y) . For complex reasoning tasks, such as mathematical theorem proving or competitive programming, this reward is typically sparse and binary, r(x,y)\in\{0,1\} . The standard RLVR objective seeks to maximize the expected reward while penalizing deviations from an initial reference policy \pi_{\text{ref}} (typically the supervised fine-tuned model) to prevent catastrophic forgetting or "over-optimization" ( 7 ) : \mathcal{J}_{\text{RLVR}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(\cdot|x)}\left[r(x,y)\right]-\beta D_{\mathrm{KL}}\left(\pi_{\theta}(\cdot\mid x)\parallel\pi_{\text{ref}}(\cdot\mid x)\right), (1) where \beta controls the strength of the KL penalty. Modern post-training pipelines often optimize this objective using algorithms like Group Relative Policy Optimization (GRPO) ( 33 ) and its