iGRPO: Self - Feedback–Driven LLM Reasoning
Ali Hatamizadeh, Shrimai Prabhumoye, Igor Gitman, Ximing Lu, Seungju Han, Wei Ping, Yejin Choi, Jan Kautz · 2026
Read iGRPO: Self - Feedback–Driven LLM Reasoning by Ali Hatamizadeh, Shrimai Prabhumoye, Igor Gitman, Ximing Lu, Seungju Han, Wei Ping, Yejin Choi, Jan Kautz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09000.txt.
Introduction
Language models still produce shaky, inconsistent math solutions, and training them better is the goal of iGRPO. Ali Hatamizadeh and seven co-authors extend Group Relative Policy Optimization with dynamic self-conditioning. Stage one samples several exploratory drafts and keeps the highest-reward one. Stage two appends that draft to the prompt and trains refinements that must beat the model's own best attempt.
Under matched rollout budgets, iGRPO consistently beats GRPO across base models. Applied to OpenReasoning-Nemotron-7B, it sets new state-of-the-art scores of 85.62 percent on AIME24 and 79.64 percent on AIME25. Ablations show the refinement wrapper generalizes beyond GRPO variants and benefits from a generative judge.
Catalog record
| CREATOR | Ali Hatamizadeh, Shrimai Prabhumoye, Igor Gitman, Ximing Lu, Seungju Han, Wei Ping, Yejin Choi, Jan Kautz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09000
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
iGRPO: Self - Feedback–Driven LLM Reasoning Ali Hatamizadeh, Shrimai Prabhumoye, Igor Gitman, Ximing Lu, Seungju Han, Wei Ping, Yejin Choi, Jan Kautz Abstract Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions. Reinforcement Learning (RL) is a framework for aligning these models with task-specific rewards, improving overall quality and reliability. Group Relative Policy Optimization (GRPO) is an efficient, value-function-free alternative to Proximal Policy Optimization (PPO) that leverages group-relative reward normalization. We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts. In Stage 1, iGRPO samples multiple exploratory drafts and selects the highest-reward draft using the same scalar reward signal used for optimization. In Stage 2, it appends this best draft to the original prompt and applies a GRPO-style update on draft-conditioned refinements, training the policy to improve beyond its strongest prior attempt. Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models (e.g., Nemotron-H-8B-Base-8K and DeepSeek-R1 Distilled), validating its effectiveness on diverse reasoning benchmarks. Moreover, applying iGRPO to OpenReasoning-Nemotron-7B trained on AceReason-Math achieves new state-of-the-art results of 85.62% and 79.64% on AIME24 and AIME25, respectively. Ablations further show that the refinement wrapper generalizes beyond GRPO variants, benefits from a generative judge, and alters learning dynamics by delaying entropy collapse. These results underscore the potential of iterative, self-feedback-based RL for advancing verifiable mathematical reasoning. \abscontent 1 Introduction Reinforcement Learning (RL) has proven to be successful in improving reasoning capabilities of LLMs by optimizing against task-specific reward signals. Early successes in this direction include RL from Human Feedback (RLHF) for aligning LLMs with human intent, most notably in InstructGPT ( Ouyang et al. 2022 ) and ChatGPT ( Achiam et al. 2023 ) , which have demonstrated that incorporating preference-based rewards can dramatically improve both the usability and correctness of model outputs. Recently DeepSeek-R1 ( Guo et al. 2025 ) proposed a distinguishing feature which is the so-called zero configuration, wherein the RL process directly enhances the base language model. This breakthrough started several efforts which were targeted at replicating DeepSeek-R1’s methodology or refining its underlying RL mechanisms ( Zeng et al. 2025 ; Yu et al. 2025 ; Liu et al. 2025 ; Cui et al. 2025 ; Hu et al. 2025 ) . Yet, in the realm of complex reasoning, RL algorithms typically do not incorporate any form of feedback or reflection on the model’s own outputs. Humans, by contrast, rarely solve nontrivial problems in a single pass: they often iterate on initial drafts, identify mistakes, and refine their solutions based on internal feedback ( Flower & Hayes 1981 ; Simon 2012 ; Braidotti 2019 ; Flavell 1979 ; Schön 2017 ; Polya 2014 ) . There is growing evidence that self-feedback mechanisms can bolster multi-step reasoning and the capacity to correct errors ( Madaan et al. 2023 ; Shinn et al. 2023 ) . However, existing RL frameworks do not capitalize on this iterative refinement process, leaving a critical gap between how humans naturally solve problems and how LLMs are typically trained to do so. In this work, we propose to fill this gap with Iterative GRPO (iGRPO) which is a powerful extension of GRPO ( Shao et al. 2024 ) . As illustrated in Figure 1 , Our method operates in two stages. First, we draw multiple candidate completions from the model and compute their relative rewards via the group-based mechanism of GRPO. We then select the highest-scoring draft and this serves as the "first-draft" output of the model. We consider this highest-scoring response as a guide to improve the final output. Hence, it is provided as a self-feedback to the model. We feed it back to the model alongside the original prompt. By conditioning on this exemplar, the second stage encourages the model to refine and surpass its own best prior attempt. Notably, this design preserves the efficiency of GRPO while introducing only minimal extra overhead, as iGRPO still relies on the same set of group-based reward signals. In doing so, iGRPO offers a promising avenue for self-guided improvement, enabling LLMs to iteratively improve their reasoning capabilities. Figure 1 : Iterative GRPO (iGRPO): During Exploratory Draft Generation , the model selects a high-scoring “best draft” from initial samples and appends it to the prompt for Conditioned Refinement . This augmented context guides the generation of new group-based updates, creating a bootstrapping effect where the policy continuously improves its own conditioning signal to enhance reasoning. We conduct a series of controlled experiments to compare iGRPO and GRPO under identical training conditions, using different base models trained on the Mathematics Aptitude Test of Heuristics (MATH) ( Hendrycks et al. 2021 ) dataset. Specifically, we evaluate DeepSeek-R1 Distilled ( Guo et al. 2025 ) and OpenMath-Nemotron ( Moshkov et al. 2025 ) on an extensive array of mathematical reasoning benchmarks, including AIME24 ( AI-MO 2024a ) , AIME25 ( OpenCompass 2025 ) , MATH500 ( Lightman et al. 2023 ) , AMC23 ( AI-MO 2024b ) , GSM8K ( Cobbe et al. 2021 ) , and Minerva Math ( Lewkowycz et al. 2022 ) . For models with 7B and 14B parameters, iGRPO consistently outperforms standard GRPO. Moreover, by leveraging iGRPO algorithm with OpenReasoning-Nemotron-7B model ( NVIDIA 2025 ) on the large-scale AceReason - Math ( Chen et al. 2025b ) dataset ( R1 2024 ) , we push the state of the art on AIME24 and AIME25 to 85.62% and 79.64%, respectively. These findings underscore the effectiveness of incorporating a self-feedback stage into group-based RL optimization, particularly for complex mathematical reasoning tasks. 2 Related Work RL for Reasoning. Reinforcement learning (RL) has become an important tool for refining large language models on logical and analytical tasks ( Lambert et al. 2024 ) . Early self-improvement lines of work, such as STaR-style bootstrapping with verified outcomes and sampling-based selection (as discussed in ( Lambert et al. 2024 ) ), demonstrate that iteratively leveraging model-generated solutions can improve reasoning behavior. More recent systems scale outcome-driven training substantially ( Jaech et al. 2024 ) , and open-weight efforts such as DeepSeek-R1 report strong reasoning performance under similar training regimes ( Guo et al. 2025 ) . Beyond natural language tasks, RL on procedurally generated puzzles ( Xie et al. 2025 ) and settings with limited human demonstrations ( Wang et al. 2025 ) further highlight the breadth of RL as a mechanism for improving mathematical and logical reasoning. GRPO and Variants. There has also been rapid progress in refining and extending GRPO for large-scale LLM training. Dr. GRPO ( Liu et al. 2025 ) analyzes sources of bias in GRPO-style token-level objectives and proposes modifications such as removing divisions by sequence length and group-level standard deviation to better match unbiased policy gradients. DAPO ( Yu et al. 2025 ) targets long chain-of-thought training through dynamic sampling, decoupled clipping, and reward shaping designed to mitigate instability and reward noise. GSPO ( Zheng et al. 2025 ) instead operates at the sequence level, redefining importance ratios and applying sequence-level clipping to improve stability, especially in Mixture-of-Experts settings. Whereas these approaches primarily focus on stabilizing or correcting the underlying optimization