DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management

Xuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, Yuting Liu · 2025

Read DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management by Xuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, Yuting Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12951.txt.

Introduction

Xuerui Su and co-authors retune the reinforcement learning that teaches language models to reason, in a 2025 paper. Modern reasoning training leans on hand-written reward rules, yet how those rules trade exploration against exploitation, and how reward spread affects learning, was little studied. Their algorithm, DGRO, splits one control knob of the standard method into two. One scales the learning signal itself. The other governs how far the model may drift from the policy that generated its samples.

That split gives precise command of the explore-exploit balance and ports to related methods. Separately, the authors show that the variance of rewards steers both how fast training converges and how good the final model is, and they manage it deliberately. On a logic benchmark the method reached 96.9 percent average accuracy, the best reported, and it generalized well across mathematics tests.

Catalog record

CREATORXuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, Yuting Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12951

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management Xuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, Yuting Liu Abstract Inference scaling further accelerates Large Language Models (LLMs) toward Artificial General Intelligence (AGI), with large-scale Reinforcement Learning (RL) to unleash long Chain-of-Thought reasoning. Most contemporary reasoning approaches usually rely on handcrafted rule-based reward functions. However, the tarde-offs of exploration and exploitation in RL algorithms involves multiple complex considerations, and the theoretical and empirical impacts of manually designed reward functions remain insufficiently explored. In this paper, we propose Decoupled Group Reward Optimization (DGRO), a general RL algorithm for LLM reasoning. On the one hand, DGRO decouples the traditional regularization coefficient into two independent hyperparameters: one scales the policy gradient term, and the other regulates the distance from the sampling policy. This decoupling not only enables precise control over balancing exploration and exploitation, but also can be seamlessly extended to Online Policy Mirror Descent (OPMD) algorithms in Kimi k1.5 and Direct Reward Optimization. On the other hand, we observe that reward variance significantly affects both convergence speed and final model performance. We conduct both theoretical analysis and extensive empirical validation to assess DGRO, including a detailed ablation study that investigates its performance and optimization dynamics. Experimental results show that DGRO achieves state-of-the-art performance on the Logic dataset with an average accuracy of 96.9%, and demonstrates strong generalization across mathematical benchmarks. 1 Introduction Inference scaling has significantly enhanced the reasoning capabilities of Large Language Models (LLMs) [ 1 , 2 , 3 , 4 ] , as exemplified by OpenAI O1 [ 5 ] and DeepSeek R1 [ 6 ] . A key factor in this progress is large-scale reinforcement learning (RL) [ 7 ] , which plays a central role in optimizing LLMs for complex reasoning tasks [ 8 , 9 , 10 ] . Modern RL algorithms like Proximal Policy Optimization (PPO) [ 11 ] often seek to maximize task-specific rewards while constraining the divergence from a reference policy, balancing performance and stability. Building upon PPO, Group Relative Policy Optimization (GRPO) [ 12 ] employs group-wise normalized rewards as advantages, and demonstrates strong performance on reasoning benchmarks [ 6 ] . Alternatively, reward optimization algorithms that directly minimize the deviation between rewards and policy log-probabilities, such as Kimi k1.5 [ 13 ] for LLM reasoning and Direct Reward Optimization (DRO) [ 14 ] for aligning agent behavior with human preferences, offer another promising direction for RL research. Despite this progress, the study of how the reward function interacts with the optimization mechanism in reasoning tasks remains limited. In typical Reinforcement Learning from Human Feedback (RLHF), a reward model is trained to serve as a reward signal at later stages of RL training [ 1 , 15 ] . Alternatively, recent works such as DeepSeek R1 [ 6 ] , Kimi k1.5 [ 13 ] , and Logic-RL [ 16 ] adopt rule-based reward functions to reduce training costs and improve generality. Although these approaches are practical, reward designs are often heuristic and lack theoretical grounding on the properties of reward functions that lead to more effective RL optimization. Moreover, the impact of RL algorithms on the exploration–exploitation trade-off in reasoning tasks remains underexplored and theoretically ungrounded. In this paper, we introduce Decoupled Group Reward Optimization (DGRO), a general RL algorithm for LLM reasoning. Specifically, DGRO decouples the regularization coefficient in the KL-regularized reward maximization objective into two independent coefficients: one controlling the strength of the policy gradient, and the other governing the distance between reference policy and actor policy. This decoupling provides finer control over both optimization dynamics and reasoning quality, enabling a more effective balance between exploration and exploitation. Theoretically, we provide theoretical insights about how the decoupled coefficients and reward variance influence reasoning performance and optimization dynamics. Moreover, we also analyze the validity and error bound of using the reward mean as an approximation for the soft value function under various conditions. These findings provide practical guidance for exploration-exploitation control and reward variance management. Our theoretical predictions are validated through experiments on the K&K and Math datasets, where DGRO achieves superior performance compared to the base model. Our main contributions are summarized as follows: 1. We propose DGRO, a general reward optimization algorithm for LLM reasoning tasks. The proposed objective extends prior methods and enables more efficient exploration–exploitation trade-offs through the decoupling of regularization coefficients. 2. We provide the theoretical analysis about the effect of the decoupled coefficients and reward variance. Appropriate combinations of decoupled coefficients and higher reward variance can facilitate faster convergence and better final accuracy. 3. We demonstrate the effectiveness of DGRO on reasoning benchmarks like Logic and Math tasks. Ablation studies further support our theoretical claims and offer practical guidance on exploration-exploitation control and reward variance management. 2 Background 2.1 Related Works RL algorithms have been widely used to improve the reasoning and human alignment abilities of LLMs [ 6 , 16 , 17 , 18 , 19 , 20 , 21 ] . Among various approaches, Proximal Policy Optimization (PPO) [ 22 , 23 ] remains a widely used RL algorithm, typically via a KL-regularized reward maximization objective. Building upon this, DeepSeek R1 [ 6 ] adopts Group Relative Policy Optimization (GRPO) [ 12 ] , which employs group-wise normalized rewards as advantages, thereby improving sample efficiency and reducing gradient variance. Rather than relying on PPO’s surrogate objective [ 23 ] , an alternative line of research reformulates the reward optimization problem itself, deriving exact closed-form updates and designing new reinforcement learning algorithms accordingly that directly minimize the deviation between rewards and policy log-probabilities [ 14 , 21 , 13 ] . For instance, Direct Reward Optimization (DRO) [ 14 ] directly approximates the reward-maximizing policy in Human Alignment tasks. And similar approach has also been used for enhancing LLM reasoning ability like Kimi k1.5 [ 13 ] , but without providing systematic theoretical analysis or guarantees. Reward modeling is important for reasoning ability. Traditional RL pipelines train reward models from data, but these are susceptible to reward hacking [ 24 , 25 ] . As a result, preference-based approaches such as Direct Preference Optimization (DPO) [ 20 ] , Identity Preference Optimization (IPO) [ 26 ] , and rule-based reward assignment [ 6 , 17 , 16 ] are introduced. Rule-based methods typically assign predefined rewards (e.g., +1 for correct reasoning chains and -1 otherwise). In addition to reward modeling, other training parameters such as response length and regularization strength have also been found to affect the reasoning quality of RL-trained models [ 27 , 28 ] . While these methods have led to significant empirical progress, most lack theoretical analysis regarding how reward formulation and optimization dynamics influence model behavior. This motivates a more principled study of reward-based RL for reasoning tasks. 2.2 Preliminary In the response reasoning process, LLM is prompted with an input question x to generate an output y , where both x,y are sequences of tokens. The goal of a reasoning model is to learn a chain-of-thought policy that produces accurate responses through structured reasoning. PPO [ 23 ] formulates the objective as maximizing the expectation of reward r_{\phi} while penalizing deviation

Read the complete text