RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization

Zhiyuan Zeng, Jiashuo Liu, Zhangyue Yin, Ge Zhang †, Wenhao Huang †, Xipeng Qiu † · 2025

Read RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization by Zhiyuan Zeng, Jiashuo Liu, Zhangyue Yin, Ge Zhang †, Wenhao Huang †, Xipeng Qiu † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04285.txt.

Introduction

Zeng Zhiyuan, Jiashuo Liu, Zhangyue Yin, Ge Zhang, Wenhao Huang, and Xipeng Qiu present RLoop, a self-improving training loop for reasoning models learned with verifiable rewards. Such training overfits: rewards climb while generalization falls, as policies over-specialize and forget diverse solutions found along the way.

RLoop explores with reinforcement learning, filters successful trajectories into an expert dataset, then fine-tunes the starting policy on that set by rejection sampling. The refined policy seeds the next round, converting passing variation into durable gains. Average accuracy rose by nine percent and a pass-at-many-tries metric by over fifteen percent against plain reinforcement learning in their tests.

Catalog record

CREATORZhiyuan Zeng, Jiashuo Liu, Zhangyue Yin, Ge Zhang †, Wenhao Huang †, Xipeng Qiu †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04285

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization Zhiyuan Zeng, Jiashuo Liu, Zhangyue Yin, Ge Zhang †, Wenhao Huang †, Xipeng Qiu † Abstract While Reinforcement Learning for Verifiable Rewards (RLVR) is powerful for training large reasoning models, its training dynamics harbor a critical challenge: “RL overfitting,” where models gain training rewards but lose generalization. Our analysis reveals this is driven by policy over-specialization and catastrophic forgetting of diverse solutions generated during training. Standard optimization discards this valuable inter-step policy diversity. To address this, we introduce RLoop , a self-improving framework built on iterative policy initialization. RLoop transforms the standard training process into a virtuous cycle: it first uses RL to explore the solution space from a given policy, then filters the successful trajectories to create an expert dataset. This dataset is used via Rejection-sampling Fine-Tuning (RFT) to refine the initial policy, creating a superior starting point for the next iteration. This loop of exploration and exploitation via iterative re-initialization effectively converts transient policy variations into robust performance gains. Our experiments show RLoop mitigates forgetting and substantially improves generalization, boosting average accuracy by 9% and pass@32 by over 15% compared to vanilla RL. 1 Fudan University 2 M-A-P 3 Shanghai Innovation Institute [email protected] [email protected] [email protected] [email protected] † † footnotetext: †Corresponding authors 1 Introduction (a) Reward on Training set. (b) Accuracy on validation set. (c) Pass@32 on validation set. Figure 1 : The reward, accuracy and pass@32 score of Qwen-2.5-math-7b trained with the DAPO algorithm evaluated on AIME-2024. Reinforcement Learning (RL), particularly through policy gradient methods such as PPO and its variants, has emerged as a cornerstone for aligning Large Language Models (LLMs) with complex human objectives. By enabling optimization for non-differentiable reward signals, RL has catalyzed significant advancements in diverse domains, including instruction following ( 20 ) and mathematical reasoning ( 6 ) . However, our investigation into the application of RL for complex reasoning tasks reveals a critical, yet previously under-explored, challenge: a phenomenon we term RL overfitting , analogous to its supervised learning counterpart. As illustrated in Figure 1 , we observe a stark divergence between the training objective and true generalization performance. While the in-distribution reward signal exhibits a steady increase throughout training (e.g., for over 700 steps), the model’s generalization capabilities—measured by out-of-distribution test accuracy and pass@k metrics—stagnate or even degrade much earlier (e.g., around 140 steps). This divergence strongly suggests that the RL agent becomes overly specialized in exploiting high-reward trajectories within its known distribution, leading to a model that is confident yet brittle when confronted with unseen problems. To dissect the underlying dynamics of this overfitting phenomenon, we conducted a deeper empirical analysis of the RL training process. As shown in Figure 2(b) , we found that standard RL training suffers from catastrophic forgetting , especially in the later stages, where the model discards approximately 30% of the knowledge acquired during early training. This finding indicates that policies at different training steps are substantially distinct. Such inter-step policy diversity represents a valuable asset for exploration, yet it is typically discarded in conventional training paradigms. While prior work has acknowledged the importance of trajectory diversity ( 5 ; 22 ) , it has primarily focused on the diversity generated by a single policy at a fixed step, overlooking the rich diversity across different training checkpoints. Inspired by the potential of harnessing this inter-step diversity, we propose RLoop , a self-improving framework centered around iterative policy initialization. Instead of a single, monolithic training run, RLoop recasts the process as a virtuous cycle where the policy is progressively refined and re-initialized. Each iteration consists of two phases: 1. Exploration Phase: Starting from the current policy \theta_{i} , we run a standard RL process not to find a single optimal policy, but to generate a diverse pool of solution trajectories. The significant policy shifts across RL steps act as a powerful, built-in exploration mechanism. 2. Exploitation Phase: We curate the trajectories generated during exploration by filtering for successful outcomes. This “expert” dataset, D_{\text{expert}}^{i} , is then used to refine the initial policy \theta_{i} via Supervised Fine-Tuning. The resulting improved policy, \theta_{i+1} , serves as a superior starting point for the subsequent exploration phase. The crucial step is that this consolidated policy \theta_{i+1} is not the final output, but serves as a superior initial policy for the next Exploration Phase. RLoop thus establishes a self-contained improvement loop: RL explores possibilities from a stable base, and RFT consolidates the findings into a better base. This process of iterative policy initialization allows the model to systematically accumulate knowledge, turning the transient diversity from RL into robust, generalizable capabilities. Unlike prior works that rely on external expert data to bridge RL and supervised learning ( 25 ; 31 ; 3 ) , RLoop bootstraps its own progress. To further stabilize this self-improvement, we incorporate an active learning strategy to ensure the model continually focuses on challenging problems. Our main contributions are summarized as follows: • We empirically identify and characterize the “RL overfitting” phenomenon in LLMs, demonstrating that reward improvements do not necessarily translate to enhanced generalization. • We reveal that this overfitting is linked to catastrophic forgetting and highlight the untapped potential of inter-step policy diversity, a valuable resource discarded by standard RL. • We propose RLoop , an iterative self-improvement framework that effectively balances exploration and exploitation by alternating between RL for diverse solution generation and RFT for knowledge consolidation. • Our experiments demonstrate that RLoop significantly outperforms vanilla RL on challenging math reasoning benchmarks, particularly in pass@k metrics. We further show that RLoop mitigates forgetting and make the RL training more stable. 2 Related Works Generalization of RLVR. The generalization capabilities of RLVR are a subject of active research with conflicting findings. Several theoretical and empirical studies suggest that RL can lead to strong generalization ( 4 ; 30 ; 1 ) , with some work even demonstrating its effectiveness with a single question-answer pair ( 23 ) . However, this optimistic view is challenged by other research. 29 empirically found that while RLVR improves greedy-decoding accuracy, it can degrade pass@k performance, especially for large k . This suggests that standard RL may merely improve test-time efficiency rather than enhancing the model’s core ability to solve novel problems. Conversely, other studies ( 17 ) report that RLVR can indeed improve pass@k scores on certain tasks. Our work contributes to this debate by identifying a key dynamic: both accuracy and pass@k can degrade during training due to an overfitting-like phenomenon, which we aim to resolve. Efforts to improve RLVR generalization can be categorized into three main perspectives: 1. Data-centric Approaches: These methods focus on enriching the training data. For instance, 13 and 15 propose augmenting the question set to expose the model to a wider range of states. Others focus on curriculum learning; 21 found that mixing simple and hard questions facilitates knowledge transfer, while 14 advocate for increasing the rollout budget for more challenging problems, based on the finding that performance correlates with the number of unique problems solved. 2. Algorithm-centric Approaches: These methods modify the RL algorithm itself. A prominent line of work explores the relationship between performance and policy entropy ( 5 ; 22

Read the complete text