EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
Xingjian Zhang, Siwei Wen, Wenjun Wu, Lei Huang · 2025
Read EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity by Xingjian Zhang, Siwei Wen, Wenjun Wu, Lei Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21848.txt.
Introduction
Xingjian Zhang and co-authors fix a quiet failure in how language models learn to reason. The popular GRPO training method scores a group of sampled answers against each other, but when every answer in the group earns the same reward, the learning signal collapses to nothing. Their 2025 paper first studies why coaxing the model to reflect does not cure the sameness, measuring response entropy at the level of single samples.
Their algorithm, EDGE-GRPO, attacks the collapse two ways. An entropy-driven term keeps the advantage signal diverse, rewarding answers that differ in informative ways. A guided error-correction step then repairs wrong attempts instead of discarding them, turning failures into training material. Across standard reasoning benchmarks, the method beat the plain algorithm and its variants. The authors argue that in group-based training, diversity of feedback is not decoration but the fuel itself.
Catalog record
| CREATOR | Xingjian Zhang, Siwei Wen, Wenjun Wu, Lei Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity Xingjian Zhang, Siwei Wen, Wenjun Wu, Lei Huang Abstract Large Language Models (LLMs) have made remarkable progress in enhancing step-by-step reasoning through reinforcement learning. However, the Group Relative Policy Optimization (GRPO) algorithm, which relies on sparse reward rules, often encounters the issue of identical rewards within groups, leading to the advantage collapse problem. Existing works typically address this challenge from two perspectives: enforcing model reflection to enhance response diversity, and introducing internal feedback to augment the training signal (advantage). In this work, we begin by analyzing the limitations of model reflection and investigating the policy entropy of responses at the fine-grained sample level. Based on our experimental findings, we propose the EDGE-GRPO algorithm, which adopts E ntropy- D riven Advantage and G uided E rror Correction to effectively mitigate the problem of advantage collapse. Extensive experiments on several main reasoning benchmarks demonstrate the effectiveness and superiority of our approach. It is available at https://github.com/ZhangXJ199/EDGE-GRPO . Introduction Recent advancements in large reasoning models, such as OpenAI-o1 ( Jaech et al. 2024 ) and Kimi-K1.5 ( Team et al. 2025 ) , have shown impressive progress in complex tasks involving mathematics and coding. Among them, the Group Relative Policy Optimization (GRPO) algorithm ( Shao et al. 2024 ) has attracted considerable attention from researchers. By discarding the value function used in the PPO algorithm ( Schulman et al. 2017 ) and instead computing rewards and relative advantages across sampled responses within each group, it significantly reduces resource consumption during training while improving reasoning performance. For the computation of rewards in the GRPO algorithm, some studies adopt a Process Reward Model (PRM) to provide more fine-grained feedback ( Cui et al. 2025 ; Wang et al. 2025b ) . However, it introduces substantial computational overhead. As a result, other works abandon the reward model in favor of using rule-based reward functions ( Zhou et al. 2025 ; Zhang et al. 2025b ) . However, this often leads to sparse rewards, where all responses within a group receive identical rewards. Consequently, the calculated advantages for each response become zero, cease to provide effective policy gradient and leading to advantage collapse during training. This phenomenon severely limits the efficiency of the sample. Figure 1: Performance comparison with other open-source models on Olympiad and Minerva. Our method achieves competitive and excellent performance with only 1K training samples. These models are all post-trained based on Qwen2.5-Math-7B. Recent studies have primarily sought to alleviate this problem from two perspectives. At the response level, efforts focus on increasing the diversity of response to prevent identical rewards across all responses, such as enforcing model reflection on incorrect answers to reduce the occurrence of uniformly incorrect outputs within a group ( Wang et al. 2025a ; Wan et al. 2025 ) . However, the extent to which reflection contributes to performance improvement remains inconclusive. At the signal level, internal feedback is introduced to augment the advantage, such as incorporating response-related semantic entropy or policy entropy into the advantage calculation ( Chen et al. 2025 ; Cheng et al. 2025 ) . However, most studies either pursue low entropy to improve accuracy or encourage high entropy to maintain exploration, lacking fine-grained modeling of the relationship between responses and their policy entropy. In this paper, we begin by analyzing the limitation of model reflection. Quantitative experiments show that responses containing self-reflection are often associated with significantly lower accuracy. Although forced reflection can help the model correct a subset of answers, its overall effectiveness remains limited. Additionally, we observe a misconception in the model’s estimation of policy entropy at the fine-grained sample level: incorrect responses do not necessarily indicate uncertainty, some of them exhibit notably lower entropy. Conversely, the model is not always confident in its correct responses, some of which display relatively high entropy. To address these issues, we propose a simple and effective EDGE-GRPO ( E ntropy- D riven GRPO with G uided E rror Correction) algorithm. At the response level, we introduce Guided Error Correction (GEC) to enhance response diversity, providing more effective guidance even when the model encounters questions beyond its current capacity. At the signal level, we compute an Entropy-Driven Advantage (EDA) that assigns higher advantages to correct responses with low entropy and lower advantages to incorrect responses with low entropy, thereby increasing the diversity of the advantage signal. These improvements significantly mitigate the problem of advantage collapse. Across multiple reasoning benchmarks, our method achieves substantial performance gains compared to the vanilla GRPO. As shown in Figure 1 , our approach reaches comparable performance to other open-source models using only 1K training samples. Our contributions can be summarized as follows: • We analyze the key challenges faced by preliminary attempts. Specifically, at the response level, prompting the model to reflect on incorrect responses has limited effectiveness. At the signal level, fine-grained sample-level policy entropy is needed to guide the augmentation of the advantage. • We propose the EDGE-GRPO algorithm. At the response level, we introduce Guided Error Correction (GEC) to overcome the limitations of the model capacity and improve response diversity. At the signal level, we compute an Entropy-Driven Advantage (EDA) to increase the diversity of the advantage signal, significantly alleviating the problem of advantage collapse. • Extensive experiments on multiple main reasoning benchmarks demonstrate that our method consistently achieving over 20% improvement in various base models, thus validating its effectiveness and superiority. Related Work Figure 2: The reflection performance of different models. Upper: For most models, the accuracy of responses that involve self-reflection is significantly lower than the overall accuracy. Left: Fine-tuning with high-quality data that includes reflection processes helps improve the accuracy of model reflection. Right: Even when the model is forced to reflect on incorrect responses, the improvement in accuracy remains limited, these results are averaged over four types of reflection prompts. Advantage Collapse. Advantage collapse is a critical limitation of the GRPO algorithm, as it severely impairs effective gradient updates. Prior approaches typically mitigate this issue through data filtering ( Yu et al. 2025 ; Meng et al. 2025 ) , by discarding samples in which all responses within a group are either entirely correct or incorrect. However, this greatly limits sample efficiency, as challenging samples can be beneficial for improving model performance. In addition, some works ( Wang et al. 2025a ) attempt to enhance response diversity by enforcing model reflection, while others ( Chen et al. 2025 ; Cheng et al. 2025 ) introduce internal feedback to strengthen the training signal. Think More or Less. There are differing views on whether model reflection truly benefits model performance. Several works ( Muennighoff et al. 2025 ; Tian et al. 2025 ) proposed adding ”wait” to chain-of-thought reasoning to encourage the model to engage in reflection, which can improve performance. VL-Rethinker ( Wang et al. 2025a ) incorporates forced reflection during the training process to enhance the slow-thinking capability of the model. Meanwhile, other researchers argue that suppressing the tokens that trigger reflection ( Liu et al. 2025a ) , encouraging the model to generate shorter responses ( Su et al. 2025 ; Fatemi et al. 2025 ) , can reduce redundant reasoning without compromising