Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Dmytro Mozolevskyi, Muayad Ali, Waseem AlShikh · 2025

Read Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning by Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Dmytro Mozolevskyi, Muayad Ali, Waseem AlShikh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24726.txt.

Introduction

Shelly Bensal and a team of co-authors present a study from twenty twenty-five on language models that get better by mulling over their own errors. When a model fails a task, it writes a note on what went wrong. It then tries the task again with that note in view. If the retry wins, the note earns a reward in training.

The loop needs no fresh fake data and no grades but pass or fail. Gains were large on math and on tool call tasks. Small tuned models beat kin ten times their size in the same family. The team frames the loop as a path to models that self improve on hard tasks with thin feedback.

Catalog record

CREATORShelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Dmytro Mozolevskyi, Muayad Ali, Waseem AlShikh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24726

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Dmytro Mozolevskyi, Muayad Ali, Waseem AlShikh Abstract We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model’s ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback. 1 Introduction Large language models (LLMs) have demonstrated impressive capabilities across a wide range of natural language processing tasks ( Zhao et al., 2025 ) , as well as in mathematics ( Ahn et al., 2024 ) , coding ( Jiang et al., 2024 ) , and reasoning ( Huang and Chang, 2023 ) . Despite these advancements however, models still have blind spots, and there is no guarantee that a model that succeeds at one task will succeed at another, even if the task is of a similar type ( Asher et al., 2023 ; Huckle and Williams, 2025 ) . The most direct way to address this problem is to retrain or fine-tune a model on data that represents the failed task, however this may not be possible if no such dataset exists. Furthermore, if the largest state-of-the-art models also struggle to complete the task, we similarly cannot use them to generate synthetic training data ( Liu et al., 2024a ) . An alternative solution is to prompt the model to explain its reasoning or self-reflect on why it failed. For example, the popular Chain-of-Thought (CoT) paradigm ( Wei et al., 2022 ) showed that models performed significantly better at arithmetic, commonsense, and reasoning tasks if they were prompted to show their reasoning in addition to simply providing a response. Self-reflection operates on a similar principle, in that if we can detect when a LLM provides an incorrect response, we can prompt it to reflect on any flaws in its reasoning and perhaps try again ( Ji et al., 2023 ; Renze and Guven, 2024 ) . The main advantage of these approaches is that they do not require any additional training data, however their effectiveness is directly tied to the effectiveness of the reasoning/reflection prompt. In this paper, we investigate the extent to which LLMs can learn to generate better self-reflections in order to self-improve on downstream tasks. More specifically, if a model fails to complete a task on its first attempt, it generates a self-reflection which it uses to make a second attempt. If the model then succeeds on its second attempt, we use reinforcement learning (RL), specifically Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) , to reward the tokens in the self-reflection, such that future self-reflections will be more effective. In this way, models can learn how to improve upon all kinds of tasks without requiring any task-specific data; they instead just optimize how to reflect on mistakes. Our main contribution is thus a novel methodology for training a model to generate better self-reflections to improve on challenging tasks in a task-agnostic way. Crucially, this method only requires a binary success/failure signal from a response verifier, which makes it well-suited to tasks where success can be easily verified. To demonstrate the efficacy of our approach, we carry out experiments on both the APIGen function calling dataset ( Liu et al., 2024b ) and the Countdown equation task introduced by Pan et al. (2025b) . 2 Related Work 2.1 Self-Reflection Self-reflection in LLMs Self-reflection, also referred to as introspection, is a metaprompting strategy in which a language model analyzes its own reasoning in order to identify and correct potential mistakes. This paradigm has gained momentum in large language model (LLM) research as a means to boost multi-step reasoning and problem-solving performance, especially in domains such as arithmetic, commonsense reasoning, and question answering ( Wei et al., 2022 ; Madaan et al., 2023 ; Renze and Guven, 2024 ; Shinn et al., 2023 ) . Typically, self-reflection involves generating an initial answer, producing natural language feedback to critique that answer, and then refining the response based on this critique. This process can be applied iteratively, often using the same model to both generate and evaluate solutions, and may include modules such as memory buffers or explicit meta-instruction guides ( Liu et al., 2025 ; Wu et al., 2025 ) . Approaches and Limitations The methodology for self-reflection in LLMs varies along several axes. Some methods apply self-correction only to failed or low-confidence queries, while others use it for every response; feedback can be provided in the form of scalar scores, external annotations, or natural language, and may be generated by humans, external models, or the LLM itself ( Bai et al., 2022 ; Peng et al., 2023 ; Yang et al., 2022 ; Pan et al., 2025c ) . While prompting LLMs to self-reflect does improve accuracy in many settings, recent work has shown that the effectiveness depends strongly on the context: challenges include the inability to reliably identify self-errors without ground-truth oracles, diminishing returns from repeated reflection, and risks of performance deterioration for easier prompts or high-performing base models ( Huang et al., 2024 ; Zhang et al., 2024 ; Kim et al., 2023 ) . In particular, self-reflection is most effective when initial accuracy is low, question difficulty is high, and external verification is available. Conversely, LLMs may sometimes fail to recognize their own mistakes but can still benefit from external feedback when such supervision exists ( Pan et al., 2025c ; Shinn et al., 2023 ) . Training-Based Methods Recent directions focus on incorporating self-improvement capabilities during model training, either by fine-tuning on self-correction trajectories or by formulating the process as a multi-turn reinforcement learning problem ( Kumar et al., 2024 ; Qu et al., 2024 ; Wu et al., 2025 ) . These training-based methods suggest that leveraging the model’s own critiques during learning yields persistent improvements—even when no test-time self-reflection is performed. However, these approaches typically rely on larger teacher models for data generation or supervision, which can be seen as a form of knowledge distillation ( Hinton et al., 2015 ) . Our Approach Building on insights from prior research, we propose correcting only failed cases identified by an external verifier, converting its binary feedback into self-reflective prompts, and training the model to use the self-reflection to succeed at the second attempt. This

Read the complete text