Expanding the Capabilities of Reinforcement Learning via Text Feedback
Yuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, Andrea Zanette, Carnegie Mellon University · 2026
Read Expanding the Capabilities of Reinforcement Learning via Text Feedback by Yuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, Andrea Zanette, Carnegie Mellon University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02482.txt.
Description
Yuda Song and 7 co-authors study text feedback as a middle-strength training signal for language models, in a 2026 arXiv paper. Reinforcement learning from binary rewards gives one bit of information per attempt, while distillation gives dense supervision but needs costly demonstrations. Written critiques sit between the two: richer than a score, cheaper than a worked solution, and already produced by users and judges everywhere.
Their setup makes feedback available during training but not at test time, so models must internalize it. Two methods do that. Self distillation trains the single-turn policy to match its own feedback-conditioned second attempts. Feedback modeling adds predicting the critique as an auxiliary task. Theory supports both methods. Experiments on reasoning puzzles, competition math, and creative writing beat strong baselines consistently.
Catalog record
| CREATOR | Yuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, Andrea Zanette, Carnegie Mellon University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 123 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02482