Expanding the Capabilities of Reinforcement Learning via Text Feedback

Yuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, Andrea Zanette, Carnegie Mellon University · 2026

Read Expanding the Capabilities of Reinforcement Learning via Text Feedback by Yuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, Andrea Zanette, Carnegie Mellon University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02482.txt.

Description

Yuda Song and 7 co-authors study text feedback as a middle-strength training signal for language models, in a 2026 arXiv paper. Reinforcement learning from binary rewards gives one bit of information per attempt, while distillation gives dense supervision but needs costly demonstrations. Written critiques sit between the two: richer than a score, cheaper than a worked solution, and already produced by users and judges everywhere.

Their setup makes feedback available during training but not at test time, so models must internalize it. Two methods do that. Self distillation trains the single-turn policy to match its own feedback-conditioned second attempts. Feedback modeling adds predicting the critique as an auxiliary task. Theory supports both methods. Experiments on reasoning puzzles, competition math, and creative writing beat strong baselines consistently.

Catalog record

CREATORYuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, Andrea Zanette, Carnegie Mellon University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE123 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02482

Browse the full library · About the Tower of Babel