MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop

Xuancheng Li, Haitao Li, Yujia Zhou, Yiqun Liu, Qingyao Ai · 2026

Read MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop by Xuancheng Li, Haitao Li, Yujia Zhou, Yiqun Liu, Qingyao Ai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22900.txt.

Introduction

Xuancheng Li and four co-authors improve how language models learn from feedback given in words. Reward-based training often gives only a single score, which is sparse and vague. On failed tries, a low score says the answer was wrong but not why it failed.

Their loop, MulFeRL, feeds verbal feedback back into training over several turns. Detailed comments can point at the exact step that went astray. Richer feedback should teach reasoning skills faster than scores alone, especially on hard problems where most first tries fail. Verbal feedback shines exactly on failed samples, where a plain score carries almost no usable teaching signal at all.

Catalog record

CREATORXuancheng Li, Haitao Li, Yujia Zhou, Yiqun Liu, Qingyao Ai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop Xuancheng Li, Haitao Li, Yujia Zhou, Yiqun Liu, Qingyao Ai Abstract Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in multiple domains, yet outcome-only scalar rewards are often sparse and uninformative—especially on failed samples, where they merely indicate failure and provide no insight on why the reasoning fails. In this paper, we investigate how to leverage richer verbal feedback to guide RLVR training on failed samples, and study how to convert such feedback into a trainable learning signal. Specifically, we propose MulFeRL ( Mul ti-turn Fe edback-guided R einforcement L earning). Concretely, MulFeRL builds on three mechanisms: (1) dynamic multi-turn regeneration guided by feedback, triggered only on failed samples, (2) two complementary learning signals for within-turn and cross-turn optimization, and (3) structured feedback injection into the model’s reasoning process. Trained on sampled OpenR1-Math, MulFeRL outperforms supervised finetuning and RLVR methods on in-domain and generalizes well out-of-domain. 1 1 1 We release our code at https://anonymous.4open.science/r/MulFeRL . 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has become a popular approach for improving reasoning capabilities in domains such as mathematics and code generation ( 27 ; 5 ; 3 ) . After a model produces an answer, a verifier provides a verifiable scalar signal (e.g., correct/incorrect), which is then used to optimize the policy ( 30 ; 7 ; 29 ) . However, as model performance increases, training that relies solely on final outcome correctness yields diminishing gains and eventually struggle to further improve the model ( 43 ; 42 ) . We argue that this bottleneck stems from the sparsity and uninformative content of scalar rewards ( 41 ; 34 ) . In early training stages or on harder problems, models often generate a large fraction of entirely incorrect solutions; under standard RLVR, failed solutions typically receive near-zero or negative rewards that merely indicate incorrectness, but provide little information about why the reasoning failed or how to improve it ( 1 ; 16 ) . Consequently, the resulting learning signal suffers from poor credit assignment: gradient estimates are noisy, training becomes sample-inefficient, and progress slows or even plateaus, ultimately limiting final performance ( 45 ; 33 ; 18 ) . In contrast, when errors occur, humans typically leverage verbal feedback to localize mistakes and obtain fine-grained guidance, helping learners reduce the gap between current performance and desired outcomes ( 9 ; 13 ) . Inspired by this, we incorporate verbal feedback into model training: beyond indicating whether an answer is correct, we use natural language to explicitly identify key errors, missing steps, and actionable suggestions for improvement, providing a substantially more informative, fine-grained learning signal for unsuccessful trajectories. This enables the model to learn from failed trajectories that would otherwise be treated as uniformly uninformative under outcome-only rewards. Recent work has begun exploring learning from verbal feedback, demonstrating that it can improve reasoning. However, it remains non-trivial to convert verbal feedback into a consistent and learnable signal. Some methods convert verbal feedback into scalar rewards for RL ( 37 ; 20 ; 12 ) . This makes optimization straightforward, but it distills the rich information in feedback into a much simpler signal, weakening fine-grained guidance about errors and fixes. Moreover, studies like ( 43 ) generate feedback broadly across sampled trajectories rather than explicitly targeting failures, so a substantial fraction of feedback and the resulting updates focus on already-correct trajectories, leaving less learning capacity for hard failure cases per update and reducing sample efficiency. Another line of work uses verbal feedback to generate revised solutions and then applies supervised finetuning on these revisions ( 8 ; 17 ) . While often effective, this setup may not explicitly train the model to use and internalize feedback, limiting how effectively it can learn from external feedback during training. Building on these observations, we focus on a central question: how can verbal feedback be transformed into a stable learning signal that continues to drive RL training progress on failed samples? Our key idea is that, during training, when a sample receives uniformly zero scalar reward and thus provides no informative learning signal, we introduce verbal feedback to guide regeneration and measure the resulting feedback-conditioned improvements, using them to construct an optimizable training objective, thereby converting verbal feedback into gradient signals. To this end, we propose MulFeRL ( Mul ti-turn Fe edback-guided R einforcement L earning), a dynamic multi-turn framework that strengthens RLVR by learning from verbal feedback on failed samples. MulFeRL builds on three key mechanisms: dynamic multi-turn feedback-driven regeneration that triggers targeted feedback only when all candidates fail and uses it to guide iterative regeneration; two complementary learning signals , with an in-turn GRPO ( 7 ) objective that learns from relative advantages among candidates within the same turn, and a cross-turn DPO ( 23 ) objective that learns a preference between the previous-turn attempt and its feedback-guided revision; and structured feedback injection that integrates feedback into the model’s reasoning process to improve its ability to use and learn from feedback. We train MulFeRL on sampled OpenR1-Math ( 39 ) and evaluate it on five challenging mathematical reasoning benchmarks. We further assess generalization on three out-of-distribution scientific and general reasoning tasks. Extensive experiments show that MulFeRL consistently outperforms prior supervised and RL-based finetuning baselines, and analyses of its dynamics and internal mechanisms further support its effectiveness. 2 Related Work Enhancing LLM Reasoning with Reinforcement Learning. In verifiable domains such as mathematics and code, RLVR typically trains LLMs with rule-based outcome signals and updates the policy through online optimization ( 27 ; 19 ) . A common foundation is Proximal Policy Optimization (PPO) ( 26 ) , whose clipped policy-gradient objective supports stable on-policy updates. Building on PPO, Group Relative Policy Optimization (GRPO) ( 7 ) is a variant tailored for efficient, large-scale reasoning post-training. Subsequent GRPO variants, such as Dr.GRPO ( 19 ) , DAPO ( 42 ) , and GSPO ( 44 ) , further improve training stability and efficiency. Meanwhile, methods like RAGEN ( 36 ) extend GRPO from single-shot rollouts to multi-turn interactive agent training with environment feedback. Despite algorithmic progress, RLVR remains limited by sparse scalar rewards: when verification fails, the signal offers little guidance on what went wrong or how to fix it, often leading to training plateaus ( 33 ; 18 ) . We address this by injecting verbal feedback during training to provide richer guidance and improve performance. Learning from Verbal Feedback. Verbal feedback can pinpoint errors and provide actionable correction cues, offering a richer signal than outcome-only supervision on failed trajectories ( 21 ; 6 ) . Prior work mainly uses feedback in two ways. One line incorporates feedback into RL-style updates, either by converting it into scalar rewards or using critique to guide one-shot revisions ( 37 ; 20 ; 12 ; 43 ) . While easy to optimize, this often compresses rich feedback into a coarse, single-step signal; moreover, when feedback is generated broadly rather than triggered by failures, updates can be wasted on already-correct trajectories, reducing sample efficiency ( 43 ) . Another line uses feedback to produce revised solutions and then applies supervised finetuning on these

Read the complete text