Reinforcement Learning for Reasoning in Large Language Models with One Training Example
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen · 2025
Read Reinforcement Learning for Reasoning in Large Language Models with One Training Example by Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20571.txt.
Introduction
Yiping Wang and colleagues show that one training example can teach a language model to reason far better. Their method applies reinforcement learning with verifiable rewards to a base math model. A single well-chosen problem lifted performance on a five hundred problem benchmark from thirty-six percent to nearly seventy-four. About nine points of that gain survived format correction.
Across six math benchmarks, average scores rose from under eighteen to nearly thirty-six percent. A twelve hundred example curated set matched, not beat, the single example, which it happened to contain. Two examples slightly exceeded both. The effect repeats across model families, sizes, and two reinforcement algorithms. Curiosities abound: skills generalize across problem categories, self-reflection grows more frequent, and test scores keep climbing after training accuracy saturates. The driver is the policy gradient itself, not grokking.
Catalog record
| CREATOR | Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20571
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning for Reasoning in Large Language Models with One Training Example Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen Abstract We show that reinforcement learning with verifiable reward using one training example ( 1-shot RLVR ) is effective in incentivizing the mathematical reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0% to 73.6% (8.6% improvement beyond format correction), and improves the average performance across six common mathematical reasoning benchmarks from 17.6% to 35.7% (7.0% non-format gain). This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6%, average: 35.9%), which contains the aforementioned example. Furthermore, RLVR with only two examples even slightly exceeds these results (MATH500: 74.8%, average: 36.6%). Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples. In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post-saturation generalization . Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., by incorporating entropy loss with an appropriate coefficient) in 1-shot RLVR training. We also further discuss related observations about format correction, label robustness and prompt modification. These findings can inspire future work on RLVR efficiency and encourage a re-examination of recent progress and the underlying mechanisms in RLVR. Our code, models, and data are open source at https://github.com/ypwang61/One-Shot-RLVR . 1 Introduction Figure 1: RLVR with 1 example ( green ) can perform as well as using datasets with thousands of examples ( blue ). Left/Right corresponds to MATH500/Average performance on 6 mathematical reasoning benchmarks (MATH500, AIME24, AMC23, Minerva Math, OlympiadBench, and AIME25). Base model is Qwen2.5-Math-1.5B. \pi_{1} and \pi_{13} are examples defined by Eqn. 2 and detailed in Tab. 2 , and they are from the 1.2k DeepScalerR subset (DSR-sub). Setup details are in Sec. 3.1 . We find that RLVR with 1 example \{\pi_{13}\} (35.7%) performs close to that with 1.2k DSR-sub (35.9%), and RLVR with 2 examples { \pi_{1},\pi_{13} } (36.6%) even performs better than RLVR with DSR-sub and as well as using 7.5k MATH train dataset (36.7%). Format reward ( gold ) (Appendix C.2.3 ) serves as a baseline for format correction. Detailed results are in Appendix C.1.1 . Additional results for non-mathematical reasoning tasks are in Tab. 1 . Recently, significant progress has been achieved in enhancing the reasoning capabilities of large language models (LLMs), including OpenAI-o1 [ 1 ] , DeepSeek-R1 [ 2 ] , and Kimi-1.5 [ 3 ] , particularly for complex mathematical tasks. A key method contributing to these advancements is Reinforcement Learning with Verifiable Reward (RLVR) [ 4 , 5 , 2 , 3 ] , which commonly employs reinforcement learning on an LLM with a rule-based outcome reward, such as a binary reward indicating the correctness of the model’s final answer to a math problem. Several intriguing empirical phenomena have been observed in RLVR, such as the stimulation or enhancement of specific cognitive behaviors [ 6 ] (e.g., self-reflection) and improved generalization across various downstream tasks [ 5 , 2 , 3 ] . Currently, substantial efforts are directed toward refining RL algorithms (e.g., PPO [ 7 ] and GRPO [ 8 ] ) to further enhance RLVR’s performance and stability [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . Conversely, data-centric aspects of RLVR remain relatively underexplored. Although several studies attempt to curate high-quality mathematical reasoning datasets [ 17 , 18 , 11 ] , there is relatively limited exploration into the specific role of data in RLVR. Thus, critical questions remain open: How much data is truly necessary? What data is most effective? How do the quality and quantity of the training data relate to observed empirical phenomena (e.g., self-reflection and robust generalization)? The most relevant study to these problems is LIMR [ 19 ] , which proposed a metric called learning impact measurement (LIM) to evaluate the effectiveness of training examples. Using the LIM score, they maintain model performance while reducing the number of training examples by sixfold. However, this study does not explore how aggressively the RLVR training dataset can be reduced. Motivated by these considerations, in this paper, we specifically investigate the following research question: "To what extent can we reduce the training dataset for RLVR while maintaining comparable performance compared to using the full dataset?" We empirically demonstrate that, surprisingly, the training dataset for RLVR can be reduced to as little as ONE example! This finding supports recent claims that base models already possess significant reasoning capabilities [ 13 , 20 , 6 , 21 ] , and further shows that a single example is sufficient to substantially enhance the base model’s mathematical performance. We refer to this setup as 1-shot RLVR . We summarize our contributions and findings below: Table 1: 1-shot RLVR with math examples \pi_{1} / \pi_{13} improves model performance on ARC, even better than full-set RLVR. Base model is Qwen2.5-Math-1.5B, evaluation tasks are ARC-Easy (ARC-E) and ARC-Challenge (ARC-C). We select the checkpoints achieving the best average across 6 math benchmarks. Dataset Size ARC-E ARC-C Base NA 48.0 30.2 MATH 7500 51.6 32.8 DSR-sub 1209 42.2 29.9 { \pi_{1} } 1 52.0 32.2 { \pi_{13} } 1 55.8 33.4 { \pi_{1},\pi_{13} } 2 52.1 32.4 • We find that selecting one specific example as the training dataset can achieve similar downstream performance to that of the 1.2k DeepScaleR subset (DSR-sub) containing that example. Specifically, this improves the Qwen2.5-Math-1.5B model from 36.0% to 73.6% on MATH500, and from 17.6% to 35.7% on average across 6 mathematical reasoning benchmarks, including non-trivial improvements beyond format correction (Fig. 1 ). Notably, these two examples are relatively easy for the base model, which can solve them with high probability without any training (Sec. 3.2.1 ). Additionally, 1-shot RLVR on math examples can improve model performance on non-mathematical reasoning tasks, even outperforming full-set RLVR (Tab. 1 ). • We confirm the effectiveness of 1(few)-shot RLVR across different base models (Qwen2.5-Math-1.5/7B, Llama3.2-3B-Instruct), models distilled from long Chain-of-Thought (CoT) data (DeepSeek-R1-Distill-Qwen-1.5B), and different RL algorithms (GRPO, PPO). • We highlight an intriguing phenomenon in 1-shot RLVR: post-saturation generalization. Specifically, the training accuracy on the single example rapidly approaches 100%, yet the model’s test accuracy continues to improve. Moreover, despite using only one training example, overfitting does not occur until after approximately 1.4k training steps. Even post-overfitting, while the model’s reasoning outputs for the training example become incomprehensible multilingual gibberish mixed with correct solutions, its test performance remains strong, and the reasoning outputs for the test examples remain human-interpretable. • In addition, we demonstrate the following phenomena: (1) 1-shot RLVR is viable for many examples in the full dataset when each example is individually used for training. We also discuss its connection