Text2Grad: Reinforcement Learning from Natural Language Feedback

Hanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang · 2025

Read Text2Grad: Reinforcement Learning from Natural Language Feedback by Hanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22338.txt.

Introduction

Hanyang Wang and co-authors turn plain language feedback into fine grained training signals for language models. Their paper from 2025 contrasts their plan with classic reward training, where one number grade hides why an answer won or lost. Text notes on flaws carry more, but past uses left the model weights untouched.

Their Text2Grad frame links each feedback phrase to the answer span it faults, turns those links into graded rewards, and trains on the spans at fault. Parts include a note and span pairing pipeline, a fine reward model that also writes its own critiques, and a span level trainer. Across summary, code, and question tasks it beat number reward and prompt only rivals.

Catalog record

CREATORHanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text2Grad: Reinforcement Learning from Natural Language Feedback Hanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang Abstract Traditional RLHF optimizes language models with coarse, scalar rewards that mask the fine-grained reasons behind success or failure, leading to slow and opaque learning. Recent work augments RL with textual critiques through prompting or reflection, improving interpretability but leaving model parameters untouched. We introduce Text2Grad , a reinforcement-learning paradigm that turns free-form textual feedback into span-level gradients . Given human (or programmatic) critiques, Text2Grad aligns each feedback phrase with the relevant token spans, converts these alignments into differentiable reward signals, and performs gradient updates that directly refine the offending portions of the model’s policy. This yields precise, feedback-conditioned adjustments instead of global nudges. Text2Grad is realized through three components: (1) a high-quality feedback–annotation pipeline that pairs critiques with token spans; (2) a fine-grained reward model that predicts span-level reward on answers while generating explanatory critiques; and (3) a span-level policy optimizer that back-propagates natural-language gradients . Across summarization, code generation, and question answering, Text2Grad consistently surpasses scalar-reward RL and prompt-only baselines, providing both higher task metrics and richer interpretability. Our results suggest that natural-language feedback can serve not only as explanations, but also as actionable training signals for fine-grained alignment. The code for our method is available at https://github.com/microsoft/Text2Grad . 1 Introduction Figure 1: Comparison of PPO and Text2Grad Free-form natural language feedback is abundant in real-world applications [ 1 ] . Users leave suggestions in reviews, developers comment on code pull requests, and customers critique responses from virtual assistants. Unlike scalar ratings or preference scores, this form of feedback is inherently rich and expressive. It not only pinpoints what is correct or incorrect in an output but also explains why, providing actionable guidance for improvement. Despite its ubiquity and usefulness, most learning paradigms fail to fully leverage human feedback. Reinforcement learning from human feedback (RLHF) has become the dominant method for aligning large language models (LLMs) with human preferences [ 2 , 3 , 4 , 5 , 6 ] . RLHF typically reduces preference comparisons to scalar rewards and optimizes policies via PPO [ 7 ] or DPO [ 5 ] . While effective for improving helpfulness and safety, this scalarization discards fine-grained, token-level signals about what was right or wrong—and where—leading to imprecise credit assignment, slower convergence, and reduced interpretability [ 8 , 9 , 10 ] . An alternative line of research maintains feedback in its natural language form. Methods such as ReAct [ 11 ] and Reflexion [ 12 ] prompt the model to reflect on its outputs [ 13 ] , generate critiques, and use them to self-correct in subsequent steps [ 14 ] . These approaches are inspired by how humans operate in open-ended tasks through reasoning, explanation, and dialogue, rather than relying on numerical rewards [ 15 , 16 , 17 ] . Natural language feedback in this context improves transparency and sometimes leads to better task performance. However, because these methods leave model parameters frozen, the feedback is not internalized, requiring repeated corrections and rendering it ephemeral [ 18 , 19 , 20 ] . In this paper, we propose Text2Grad , a novel framework that transforms free-form textual feedback into actionable gradients for policy optimization. As shown in Figure 1 , unlike prior work that compresses feedback into scalar rewards or applies textual critiques only at inference time, Text2Grad brings feedback into the training loop. Given a human or programmatic critique, our method aligns feedback clauses with relevant output token spans, converts these alignments into span-level reward signals, and computes a natural language gradient. This gradient is then used to perform policy updates that precisely adjust the parts of the model responsible for the error. The result is more targeted, efficient, and interpretable learning. Text2Grad is built on a complete pipeline for learning from text. First, we construct a high-quality annotation pipeline that uses GPT-4o to label model outputs with textual feedback and span-level critiques, following recent work on automated feedback generation [ 21 , 22 ] . Second, we train a unified reward model inspired by generative reward modeling [ 23 ] that jointly generates natural language critiques and structured span-level reward maps in a single autoregressive sequence. Third, we apply span-level policy optimization using a variant of PPO that integrates these fine-grained reward signals, drawing on advances in token-aware credit assignment [ 24 ] and text-based gradients [ 25 ] . We evaluate Text2Grad across diverse domains including summarization [ 26 ] , code generation [ 27 ] , and open-domain question answering [ 28 ] . Our results demonstrate that by converting language into gradients, Text2Grad not only achieves superior performance but also offers enhanced interpretability and sample efficiency, establishing natural language feedback as a powerful, direct training signal. These results suggest that natural language feedback can be more than an interpretability tool: It can be converted into principled gradients to train more capable and aligned models. In general, this paper makes the following contributions. • We formulate the problem of learning from natural language feedback via gradient-based optimization, and present Text2Grad as the first complete framework to address it. • We develop a scalable annotation pipeline and a unified reward model that together produce span-level rewards and explanatory critiques, yielding interpretable, span-level supervision . • We show that Text2Grad outperforms strong scalar-reward-based and prompt-based baselines in summarization, code generation, and question-answering benchmarks. Text2Grad demonstrates that natural language feedback, when properly aligned and grounded, can serve as a direct training signal rather than just auxiliary guidance, opening a new path for building language models that learn from human-like supervision. 2 Related Work RLHF with scalar rewards Reinforcement learning from human feedback replaces supervised labels with a reward model trained on pairwise human preferences [ 29 , 3 ] . The reward is a single scalar, and policy optimization methods such as PPO and DPO update the language model toward higher scores [ 7 , 5 ] . This recipe has advanced instruction following, safety, and summarization; a 1.3B InstructGPT model aligned in this way outperformed 175B GPT 3 on adherence and toxicity [ 3 , 4 , 2 ] . Subsequent work studies reward hacking and data noise [ 30 , 31 , 32 ] . Despite these successes, scalar rewards collapse multidimensional critiques into a single number, obscure the location of an error, and necessitate careful regularization, such as Kullback-Leibler penalties, to remain stable [ 10 , 9 ] . Even Process Reward Models (PRMs) [ 33 ] , which offer finer credit assignment, still rely on scalar signals and lack the explanatory power of natural language feedback. Recent work addresses credit assignment through span-level optimization: MA-RLHF improves efficiency via macro-action abstraction [ 34 ] , SCAR decomposes scalar rewards using Shapley-value allocation [ 35 ] , and Beyond Sparse Rewards generates intermediate numeric rewards via auxiliary language models [ 36 ] . Natural language feedback at inference time A complementary line of research keeps feedback in natural language but applies it only while the model is running. ReAct interleaves chain of thought reasoning with tool use to refine answers in question

Read the complete text