Learning from Failures in Multi-Attempt Reinforcement Learning

Stephen ChungDualityRL, Wenyu DuDualityRLJie FuShanghai AI Lab · 2025

Read Learning from Failures in Multi-Attempt Reinforcement Learning by Stephen ChungDualityRL, Wenyu DuDualityRLJie FuShanghai AI Lab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04808.txt.

Introduction

Stephen Chung, Wenyu Du, and Jie Fu train language models to learn from failed tries in this 2025 paper. Standard training gives a model one shot per question, but people improve by revising after feedback, and the authors test whether models can learn the same skill.

In their multi-attempt setup, the model answers, gets feedback on wrong answers, and tries again. A small model trained this way rose from 45.6 percent accuracy with one try to 52.5 percent with two tries on a math benchmark. A model trained the standard way barely gained from extra tries. Multi-attempt training also edged ahead on single try scores.

Catalog record

CREATORStephen ChungDualityRL, Wenyu DuDualityRLJie FuShanghai AI Lab
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning from Failures in Multi-Attempt Reinforcement Learning Stephen ChungDualityRL, Wenyu DuDualityRLJie FuShanghai AI Lab Abstract Recent advancements in reinforcement learning (RL) for large language models (LLMs), exemplified by DeepSeek R1, have shown that even a simple question-answering task can substantially improve an LLM’s reasoning capabilities. In this work, we extend this approach by modifying the task into a multi-attempt setting. Instead of generating a single response per question, the model is given multiple attempts, with feedback provided after incorrect responses. The multi-attempt task encourages the model to refine its previous attempts and improve search efficiency. Experimental results show that even a small LLM trained on a multi-attempt task achieves significantly higher accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM trained on a standard single-turn task exhibits only a marginal improvement, increasing from 42.3% to 43.2% when given more attempts during evaluation. The results indicate that, compared to the standard single-turn task, an LLM trained on a multi-attempt task achieves slightly better performance on math benchmarks while also learning to refine its responses more effectively based on user feedback. 1 Introduction Recent advancements in large-scale post-training of reinforcement learning (RL) for large language models (LLMs) have demonstrated a promising approach to enhancing their reasoning capabilities. These improvements have led to emergent abilities such as self-correction and self-refinement [ 1 , 2 ] . Most existing methods rely on single-turn tasks, where the model receives a reward based on the correctness of its single response to a question. However, single-turn tasks may be inefficient due to sparse rewards, and they do not require the LLM to learn how to respond to user feedback. In this work, we propose a simple yet effective multi-turn task that enables LLMs to learn reasoning through RL. 1 1 1 Full code is available at https://github.com/DualityRL/multi-attempt . Instead of requiring the LLM to provide a single response to a given question, we propose a multi-attempt task that allows the LLM to generate multiple responses based on feedback. Specifically, we first randomly sample N as the number of remaining attempts for each question. The model initially generates a response to a given question as usual. If the response is correct or there are no remaining attempts (i.e., N\leq 1 ), the dialogue ends. However, if the response is incorrect and there are remaining attempts (i.e., N>1 ), we provide feedback indicating that the answer is incorrect and prompt the LLM to try again, while decrementing the remaining attempts N by 1. An illustration is shown in Figure 2 , and an example dialogue from a trained LLM is shown in Figure 3 . Figure 1: Evaluation accuracy as a function of the number of allowed attempts during evaluation, averaged across five benchmarks: AIME 2024, MATH 500, AMC 2023, Minerva Math, and OlympiadBench. Both LLMs are based on Qwen 2.5 Math 1.5B and fine-tuned via RL on a small math dataset in either multi-attempt tasks or single-turn tasks (baseline). Our training pipeline is simple and applies standard RL to the multi-attempt task on a math problem dataset, largely similar to how DeepSeek R1 Zero [ 2 ] is trained. In the multi-attempt task, a reward of +1 is given if the answer is correct in any attempt, -0.5 if the answer is incorrect but in the correct format, and -1 otherwise. We use standard Proximal Policy Optimization (PPO) [ 3 ] as the training algorithm. In addition, the multi-attempt task enables the LLM to learn how to respond to user feedback through RL, rather than merely answering the initial question. In most current LLMs, the ability to respond to user feedback is generally trained via RLHF [ 4 ] or supervised fine-tuning (SFT). Compared to RLHF or SFT, learning to respond to user feedback through pure RL may lead to more interesting emergent capabilities, such as significantly improved solution refinement, as demonstrated by our experimental results. An interesting phenomenon observed in DeepSeek R1 Zero is the emergence of the Aha Moment , which refers to the model’s ability to recognize its mistakes (self-verification) and correct its answers (self-refinement). In DeepSeek R1 Zero, both self-verification and self-refinement capabilities emerge spontaneously within the model. Our multi-attempt approach facilitates the emergence of self-refinement, as the model has a strong incentive to refine its previous failed attempts (e.g., it does not receive any reward for repeating the same failed attempt). As such, multi-attempt tasks may enable the Aha Moment to emerge more easily, as the model is explicitly trained for self-refinement. Our experiments show that even a small LLM, such as a 1.5B model, can effectively learn self-refinement capabilities. As illustrated in Figure 1 , the evaluation accuracy of an LLM trained on a multi-attempt task improved from 45.6% to 52.5% on math benchmark when increasing the number of attempts from 1 to 2. In contrast, the same model trained on a single-turn task showed only a marginal gain, from 42.3% to 43.2%. We also observe that even under the standard 1-attempt evaluation, the multi-attempt LLM slightly outperforms its single-turn counterpart, highlighting the benefits of multi-attempt training. As RL gains popularity in LLMs for enhancing reasoning capabilities, we believe that a multi-turn setting, in general, provides a richer learning signal and better enhances capabilities compared to a single-turn setting. We foresee that more sophisticated capabilities could emerge in LLMs trained in a multi-turn setting via RL. Figure 2: Illustration of the multi-attempt question-answer task. We extend the single-turn question-answer task from DeepSeek R1 to a multi-attempt setting, enabling iterative refinement. Example Dialogue: System: A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within <think> </think> and the answer is enclosed within <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer> . User: You have 2 attempts to answer the question. Your final answer will be extracted automatically by the boxed tag. This is the problem: What is the value of c if x\cdot(3x+1)<c if and only if x\in\left(-\frac{7}{3},2\right) ? Assistant: <think> To solve this problem, we need to determine the value of c such that the inequality x\cdot(3x+1)<c holds true if and only if x\in\left(-\frac{7}{3},2\right) . Let’s break it down step-by-step: 1. Understanding the Given Condition: - We need to find c such that x\cdot(3x+1)<c within the given interval. 2-4. (Omitted for brevity) 5. Conclusion: - Both \frac{140}{3} and 14 are needed to be evaluated accurately. Checking both values: - \frac{140}{3}\approx 46.67 - 14 is less than \frac{140}{3} . Thus, the highest value that fits our condition is c=\frac{140}{3} . </think> <answer> \boxed{\frac{140}{3}} </answer> User: Your previous answer is wrong. Try alternatives and refine your answer. You have 1 attempt left. Assistant: <think> I apologize for the mistake. Let’s re-evaluate the problem step-by-step after rechecking our previous work. 1. Understanding the Given Condition: - We need to find c such that x\cdot(3x+1)<c within the given interval. 2-4. (Omitted for brevity) 5. Conclusion: - The correct maximum value is actually

Read the complete text