Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs

Mengqi Liao, Xiangyu Xi, Ruinian Chen, Jia Leng, Yangen Hu, Ke Zeng, Shuai Liu, Huaiyu Wan · 2025

Read Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs by Mengqi Liao, Xiangyu Xi, Ruinian Chen, Jia Leng, Yangen Hu, Ke Zeng, Shuai Liu, Huaiyu Wan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18573.txt.

Introduction

Mengqi Liao and seven co-authors fix two quiet inefficiencies in how language models are trained by reinforcement. Their 2025 paper notes that standard training gives every question the same number of attempts, called rollouts. Easy questions waste theirs, since the model already answers them, while hard questions starve, rarely stumbling onto a correct answer to learn from. Training also sharpens answers at the cost of curiosity, narrowing the model's search below its starting breadth.

Their fix allocates the rollout budget by difficulty, spending attempts where they teach something. A second mechanism adjusts the sampling temperature on the fly to hold the model's entropy steady, preserving exploration while precision improves. Together the two aim at a learner that gets more accurate without getting more blinkered. Code and data are public, and the authors frame the method as a general upgrade to the current training recipe.

Catalog record

CREATORMengqi Liao, Xiangyu Xi, Ruinian Chen, Jia Leng, Yangen Hu, Ke Zeng, Shuai Liu, Huaiyu Wan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18573

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs Mengqi Liao, Xiangyu Xi, Ruinian Chen, Jia Leng, Yangen Hu, Ke Zeng, Shuai Liu, Huaiyu Wan Abstract Reasoning large language models (LLMs) excel in complex tasks, which has drawn significant attention to reinforcement learning (RL) for LLMs. However, existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. This inefficiency stems from the fact that training on simple questions yields limited gains, whereas more rollouts are needed for challenging questions to sample correct answers. Furthermore, while RL improves response precision, it limits the model’s exploration ability, potentially resulting in a performance cap below that of the base model prior to RL. To address these issues, we propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. Additionally, we introduce an adaptive dynamic temperature adjustment strategy to maintain the entropy at a stable level, thereby encouraging sufficient exploration. This enables LLMs to improve response precision while preserving their exploratory ability to uncover potential correct pathways. The code and data is available on: https://github.com/LiaoMengqi/E3-RL4LLMs 1 Introduction Large language models (LLMs) have gained considerable attention for their capabilities across a wide range of applications Kumar 2024 . Recently, advanced reasoning models trained with reinforcement learning (RL) , such as DeepSeek-R1 Guo et al. 2025 and Kimi k1.5, Team et al. 2025 have demonstrated remarkable improvements in complex tasks like mathematics and coding, further intensifying research interest in RL for LLMs. After that, many works related to reinforcement learning for LLMs emerged Xu et al. 2025 ; Yu et al. 2025 ; Liu et al. 2025 . Among them, the most common combination is to use the GRPO Shao et al. 2024 algorithm or its variants combined with rule-based rewards for reinforcement learning. However, rule-based rewards result in very sparse reward signals. When the training data is particularly challenging, the policy struggles to sample the correct answer, causing the advantage in the GRPO algorithm to become zero. In such cases, the policy fails to obtain an update gradient. DAPO Yu et al. 2025 filters out samples with a within-group reward standard deviation of zero to avoid zero advantage and employs multiple rounds of online sampling until enough experience is gathered for a single update. This approach is highly inefficient because the cost of sampling experiences is extremely high, yet this method results in a substantial waste of rollouts. What’s more, Yue et al. 2025 highlights that during evaluation, while the RL model surpasses the base model with a small sample size (small k ), the base model achieves superior pass@ k performance as the sample size increases (large k ). This occurs because reinforcement learning prioritizes maximizing rewards, leading the model to focus probabilities on high-reward paths, potentially neglecting diverse correct answers. While encouraging more exploration during training could potentially mitigate this issue, our experiments reveal that combining entropy regularization Mnih et al. 2016 ; Williams 1992 —the widely adopted approach for fostering exploration in deep reinforcement learning—with sparse rule-based rewards can degrade performance, especially when training with challenging questions, and may even result in model collapse. To address the aforementioned issues, we first introduce a dynamic rollout budget allocation mechanism to enhance the training efficiency of RL. For simple questions that the model can answer proficiently, we reduce their rollout budget, as performing reinforcement learning on such problems yields minimal gains. The saved rollout budget is reallocated to more challenging problems, thereby increasing the likelihood of obtaining correct answers. Additionally, to promote exploration without introducing harmful gradients, we propose a temperature scheduler that dynamically adjusts the temperature to maintain a stable policy entropy level, thereby enabling more extensive exploration during training. An annealing mechanism is further integrated to effectively balance exploration and exploitation. In summary, our contributions are as follows: • We propose a dynamic rollout budget allocation mechanism that enables a more rational distribution of computational resources, allowing RL to be conducted more efficiently. • We introduce a temperature scheduler that dynamically adjusts the sampling distribution’s temperature, maintaining the entropy at a stable level to encourage more exploration. • Experimental results demonstrate that our method improves the 7B model’s pass@1 by 5.31% and pass@16 by 3.33% on the AIME 2024 benchmark compared to train with GRPO only, and consistently outperforms GRPO in pass@16 across various benchmarks. 2 Related Work Reinforcement Learning for LLMs. Ouyang et al. 2022 trains a reward model using preference data and employs Proximal Policy Optimization (PPO) Schulman et al. 2017 to perform reinforcement learning on LLMs for alignment with human preferences. Subsequently, many new approaches have employed reinforcement learning to align LLMs with human preferences Wang et al. 2024 . DeepSeekMath Shao et al. 2024 proposed the GRPO algorithm, which simplifies the training process of reinforcement learning and significantly enhances the performance of LLMs in the mathematical domain through RL. Subsequently, DeepSeek-R1 Guo et al. 2025 and Kimi k1.5 Team et al. 2025 successfully demonstrated the substantial impact of reinforcement learning combined with rule-based reward sets in enhancing the reasoning capabilities of models. Liu et al. 2025 and Yu et al. 2025 , among others, further introduced improvements to optimization algorithms to enhance training effectiveness. Exploration and Exploitation in Reinforcement Learning. Balancing exploration and exploitation is a central challenge in reinforcement learning. Common strategies include \varepsilon -greedy, Upper Confidence Bounds (UCB), and Boltzmann Exploration Sutton et al. 1998 . Boltzmann Exploration selects actions based on a softmax probability distribution proportional to the exponential of the estimated values of actions, regulated by a temperature parameter \tau . Similarly, LLMs generate tokens using a softmax distribution. Asadi and Littman 2017 introduced the Mellowmax operator to enhance the stability of Softmax, while Kim and Konidaris 2019 applied meta-gradient reinforcement learning to dynamically adjust temperature parameter of Mellowmax for better exploration-exploitation trade-offs. Moreover, Entropy Regularization, a common technique in deep reinforcement learning, adds an entropy term to the optimization objective to encourage stochastic policies and broader exploration Williams 1992 . 3 Preliminary 3.1 Group Relative Policy Optimization (GRPO) We utilize the GRPO Shao et al. 2024 algorithm to optimize the policy \pi_{\theta} (LLMs). GRPO estimates the advantage in a group-relative manner. Specifically, given a question-answer pair (q,a)\sim\mathcal{D} , the old policy \pi_{\theta_{\text{old}}} generates G individual responses \{o_{i}\}_{i=1}^{G} and then optimizes the policy model by maximizing the following objective: \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)= \displaystyle\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)} \displaystyle\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\Big(\min\Big(r_{i,t}(\theta)\hat{A}_{i,t}, \displaystyle\,\text{clip}(r_{i,t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{i,t}\Big) \displaystyle-\beta D_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})\Big)\Bigg], (1) where r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i},<_{t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i},<_{t})} and the advantage \hat{A}_{i,t} is computed as: \hat{A}_{i,t}=\frac{r_{i}-\text{mean}(\{r_{j}\}_{j=1}^{G})}{\text{std}(\{r_{j}\}_{j=1}^{G})}. (2) Here, r_{i} is the reward of response o_{i} . The term D_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}}) represents the KL divergence penalty, which is used to prevent the policy from deviating excessively from the initial policy \pi_{\text{ref}} . Following prior work Yu et al. 2025 ; Liu et al. 2025 , we do not employ this penalty term. Therefore, we do not elaborate further on this detail. 4 Methodology In this section, we first introduce our method for modeling question difficulty and propose a dynamic rollout budget allocation mechanism to allocate budgets based on question difficulty, improving training efficiency and the model’s ability to answer complex questions. Next, we introduce a temperature scheduler to maintain the policy entropy, enhancing exploration, and further

Read the complete text