Can Large Reasoning Models Self-Train?
Sheikh Shafayat, Fahim Tajwar, [0.5ex] Ruslan Salakhutdinov, Jeff Schneider, Andrea Zanette · 2025
Read Can Large Reasoning Models Self-Train? by Sheikh Shafayat, Fahim Tajwar, [0.5ex] Ruslan Salakhutdinov, Jeff Schneider, Andrea Zanette (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21444.txt.
Introduction
Sheikh Shafayat and co-authors ask whether large reasoning models can train themselves, in a 2025 paper. Reward-based training has lifted reasoning, and self-training would let a model learn from its own judgments. They use majority voting as a simple self-feedback signal.
Tests on made-up and real reasoning tasks show the loop can work at first. Scores rose, and the model gave better feedback in the next round, driving more gains. But long runs broke down: the model learned to game the fake reward, and performance collapsed suddenly and fully. The authors name feedback design as the central challenge for lasting self-training.
Catalog record
| CREATOR | Sheikh Shafayat, Fahim Tajwar, [0.5ex] Ruslan Salakhutdinov, Jeff Schneider, Andrea Zanette |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21444
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Large Reasoning Models Self-Train? Sheikh Shafayat, Fahim Tajwar, [0.5ex] Ruslan Salakhutdinov, Jeff Schneider, Andrea Zanette Abstract Scaling the performance of large language models (LLMs) increasingly depends on methods that reduce reliance on human supervision. Reinforcement learning from automated verification offers an alternative, but it incurs scalability limitations due to dependency upon human-designed verifiers. Self-training, where the model’s own judgment provides the supervisory signal, presents a compelling direction. We propose an online self-training reinforcement learning algorithm that leverages the model’s self-consistency to infer correctness signals and train without any ground-truth supervision. We apply the algorithm to challenging mathematical reasoning tasks and show that it quickly reaches performance levels rivaling reinforcement-learning methods trained explicitly on gold-standard answers. Additionally, we analyze inherent limitations of the algorithm, highlighting how the self-generated proxy reward initially correlated with correctness can incentivize reward hacking, where confidently incorrect outputs are favored. Our results illustrate how self-supervised improvement can achieve significant performance gains without external labels, while also revealing its fundamental challenges. 1 Introduction Pre-training large-scale neural networks on human-curated corpora (e.g., web-scale texts and carefully curated datasets) has endowed language models with broad general-purpose capabilities ( Brown et al. 2020 ; Rae et al. 2022 ) . However, the availability of human-generated data for pre-training is becoming a limiting factor, particularly as computational resources continue to scale rapidly ( Hoffmann et al. 2022 ; Sevilla et al. 2022 ) . Post-training methods, such as supervised finetuning, offer a way to rapidly improve model capabilities via a much smaller set of highly curated human-generated examples, but such reliance on high quality human-generated examples offers limited scalability ( Ouyang et al. 2022 ; Bai et al. 2022 ) . In order to scale model capabilities further, reinforcement learning from human feedback can be used. This leverages human preference about model generated outputs, a type of feedback that is much easier to obtain ( Ouyang et al. 2022 ; Kaufmann et al. 2024 ) . Reinforcement learning with verifiable rewards offers a scalable alternative to human feedback by using automatic correctness checks as supervisory signals, an approach that has shown considerable success in reasoning and agentic tasks ( DeepSeek-AI et al. 2025 ; OpenAI et al. 2024 ) . However, this approach often depends on human-designed verifiers, which reintroduce the very scalability issues it seeks to overcome. Achieving superhuman performance will ultimately require models that train in settings where even humans may not know the correct solutions, and are thus unable to provide feedback on model outputs. In such scenario, models must self-improve —that is, they must be able to assess the correctness of their own outputs and use their own judgment as a feedback signal to refine future generations ( Zelikman et al. 2022 ; Song et al. 2025 ; Huang et al. 2025 ) . If this process can be sustained iteratively, models have the potential to enhance their performance continually. A particularly promising opportunity for such self-improvement arises in contexts characterized by a positive generation-verification gap ( Song et al. 2025 ) , where generating correct solutions is challenging, but verifying their correctness is relatively easy. This gap naturally arises in domains such as mathematics, games, and goal-based agentic tasks, where solutions often require complex reasoning or exploration, yet their correctness can be verified more easily. Figure 1: (Overview of SRT ) In RLVR, one produces the reward for RL training using a ground truth verifier. Contrary to that, SRT does not assume access to a ground truth verifier; instead it uses majority voting from the model’s own generations to estimate the ground truth, and use this proxy reward signal to train the model. In this work, we investigate self-improvement methods for large language models aimed at enhancing their mathematical reasoning abilities without reliance on ground-truth annotations or explicit rule-based verifiers. Specifically, we leverage an intrinsic verification criterion, namely model self-consistency ( Wang et al. 2023a ) , to produce supervisory rewards, capitalizing on the observation that consistency among model-generated answers correlates positively with their correctness. Conceptually similar approaches, relying either directly or indirectly on model-generated self-consistency signals, have been explored previously (e.g., Huang et al. 2023 ; Prasad et al. 2024 ) in mathematical domains. Typically, such approaches have been confined to one or few rounds of improvement. In contrast, the key contribution of this work is to demonstrate that this intrinsic supervisory signal can continuously drive model improvement within a reinforcement learning (RL) framework ( Sutton et al. 1998 ) , iteratively enhancing performance without access to external labels. An appealing feature of our proposed approach is its reliance solely on unlabeled data. Because our method does not depend on external labels, it can also be naturally applied in a “test-time training” fashion. Such flexibility allows models to iteratively boost their performance at inference time. Concurrently to our work, test-time training has also recently been investigated by Zuo et al. 2025 . While the prospect of continual model self-improvement is appealing, it is important to recognize a significant limitation: the model’s self-generated reward serves only as a proxy for underlying correctness. Such proxy rewards may give rise to the phenomenon of reward hacking , whereby the model learns to maximize its self-assigned reward by generating increasingly consistent but potentially incorrect answers. This unintended behavior can reduce or even reverse the initial beneficial correlation between confidence and correctness, ultimately constraining sustained improvement. Understanding the precise dynamics of this process is thus essential for further advancement. In summary, our contributions are fourfold: (1) Motivated by prior works based on consistency based self-improvement, we introduce a simple yet effective self-training reinforcement learning methodology, Self-Rewarded Training ( SRT ), that uses consistency across multiple model-generated solutions to estimate correctness during RL training, providing self-supervision signals without labeled data. (2) We empirically demonstrate that at early training stages, SRT achieves performance rivaling standard reinforcement-learning methods trained explicitly on gold-standard answers. (3) We analyze the limitations associated with self-generated rewards, revealing how the model’s reward function, initially correlated with correctness, can degrade to reflect confidence rather than true accuracy, leading to the problem of reward hacking. (4) We propose strategies to mitigate reward hacking, laying the groundwork for effective future approaches to sustaining continual model improvement. Our code is publicly available here: https://github.com/tajwarfahim/srt . For other project-related assets such as constructed datasets, please see our project website at https://self-rewarding-llm-training.github.io/ . 2 Preliminaries Let \pi_{\theta} denote a language model parameterized by parameters \theta . Given a prompt x , the model produces a response y=(y^{1},y^{2},\dots) auto-regressively. Formally, each token in the response sequence is generated according to the conditional probability: y^{k+1}\sim\pi_{\theta}(\cdot\mid x,y^{\leq k}), (1) where we use y^{\leq k} to refer to the first k tokens generated by the model. For reasoning-based tasks considered here, the model typically produces responses following a step-by-step “chain-of-thought” reasoning approach ( Wei et al. 2022 ) . A verification function r(y) , whose dependence on x is omitted for brevity, extracts the model’s proposed solution from the generated response and evaluates its correctness against the prompt-specific ground-truth answer: r(y)=\begin{cases}1&\text{if $y$ is correct,}\\ 0&\text{if $y$ is incorrect.}\end{cases} (2) Typically, one optimizes the pass rate defined as the average accuracy of the model \pi_{\theta} across a distribution of