On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence

Lei Pang, Ruinan Jin · 2025

Read On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence by Lei Pang, Ruinan Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02833.txt.

Introduction

Lei Pang, Jun Luo, and Ruinan Jin present a corrected version of a popular method for training language models from human feedback. Their 2025 paper studies GRPO, which trains without a separate value model by centering rewards inside each group of sampled answers. Small design choices in that update can distort learning.

Their version, TIC-GRPO, caps the importance ratio and uses a ratio built from the whole response, not single tokens. The authors trace how length handling inflates certain error terms in the older method. Their bounds show a cleaner link between response length and update noise. Tests on two sizes of a known model family, across reasoning and coding benchmarks, check the claims. The work aims at training that is both provably steadier and fast to converge.

Catalog record

CREATORLei Pang, Ruinan Jin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence Lei Pang, Ruinan Jin Abstract Group Relative Policy Optimization (GRPO), recently proposed by DeepSeek, is a critic free reinforcement learning algorithm for fine tuning large language models. It replaces the value function in Proximal Policy Optimization (PPO) with group normalized rewards, while retaining PPO style token level importance sampling based on an old policy. We show that GRPO update rule in fact estimates the policy gradient at the old policy rather than the current one. However, since the old policy is refreshed every few steps, the discrepancy between the two remains small limiting the impact of this bias in practice. We validate this through an ablation study in which importance sampling is entirely removed, and updates are instead performed using the gradient estimated at a fixed old policy across multiple optimization steps. Remarkably, this simplification results in performance comparable to standard GRPO. Motivated by these findings, we propose a new algorithm: Trajectory level Importance Corrected GRPO (TIC GRPO). TIC GRPO replaces token level importance ratios with a single trajectory level probability ratio, yielding an unbiased estimate of the current policy gradient while preserving the critic free structure. Furthermore, we present the first theoretical convergence analysis for GRPO style methods, covering both the original GRPO and our proposed variant. 1 Introduction Reinforcement learning from human feedback (RLHF) has become a standard technique for aligning large language models (LLMs) with desired behaviors. Among RLHF methods, Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) is widely adopted but requires training an additional value network (critic), making it resource-intensive and difficult to scale. To address this, recently proposed Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) , a critic-free alternative that estimates advantages via group-wise reward normalization while retaining PPO-style importance sampling over an old old policy. GRPO has since been deployed in several open-source RLHF pipelines due to its simplicity and effectiveness. Despite its empirical success, the theoretical properties of GRPO remain underexplored. In particular, GRPO employs token-level importance sampling using the old policy, but the update rule is not a direct estimator of the current policy gradient. In this work, we show that the practical GRPO update actually corresponds to the policy gradient evaluated at the old policy \pi_{{\text{old}}} , plus a bias term that arises due to the mismatch between \pi and \pi_{{\text{old}}} . However, this bias remains small in practice, as \pi_{{\text{old}}} is typically refreshed by relpaced it by current policy \pi every 4–10 optimization steps. This leads to only limited divergence between the current and old policies, a finding we confirm through an ablation study: we entirely remove importance sampling and, in each inner loop where the old policy \pi_{\text{old}} is held fixed before being refreshed to the current policy \pi , perform all updates using gradients estimated at \pi_{\text{old}} . Despite this simplification, the resulting performance remains comparable to that of standard GRPO. Motivated by this analysis, we propose a new algorithm: Trajectory-level Importance-Corrected GRPO (TIC-GRPO) . Instead of computing token-level importance ratios, TIC-GRPO uses a single trajectory-level probability ratio between the current and old policy, yielding an unbiased estimator of the current policy gradient while maintaining GRPO’s memory efficiency. Furthermore, we present the first theoretical convergence analysis for GRPO-style methods, covering both the original GRPO and our proposed variant. We validate TIC-GRPO on two standard alignment benchmark AIME. Our experiments show that TIC-GRPO significantly outperforms standard GRPO in both accuracy and convergence speed. We also release training logs and commit history verifying that our work was completed independently prior to a concurrent similar study. Overall, TIC-GRPO offers both theoretical clarity and practical improvements for scalable reinforcement learning fine-tuning of LLMs. Related Work A recent concurrent work by https://arxiv.org/abs/2507.18071 proposes a similar idea of replacing token-level importance sampling in GRPO with a trajectory-level formulation. Our work was developed independently and completed concurrently; see Appendix A for supporting evidence. In comparison, our work offers a more detailed explanation of why the original GRPO update remains effective in practice despite its inherent bias, which we attribute to the limited policy drift arising from frequent updates to the old policy. Moreover, we present the first theoretical convergence analysis for GRPO-style methods, covering both the original GRPO and our proposed trajectory-level variant. Contributions This paper makes the following key contributions: • We analyze the practical update rule of GRPO and show that it estimates the policy gradient at the old policy \pi_{\text{old}}, not the current one. We further explain why this approximation remains effective in practice due to limited policy drift. • We propose a new algorithm, TIC-GRPO, which replaces token-level importance sampling with a single trajectory-level ratio. This yields an unbiased estimate of the current policy gradient while preserving GRPO’s efficiency. • We provide the first theoretical convergence analysis for GRPO-style methods, including both the original GRPO and our variant. • We empirically validate TIC-GRPO on AIME dataset, demonstrating consistent improvements in accuracy and convergence speed over the original GRPO. Our training logs show that this work was completed independently prior to a concurrent study proposing a similar idea. 2 Preliminaries: Reinforcement Learning for LLMs and GRPO We begin by formalizing the reinforcement learning (RL) setup used for large language model (LLM) alignment, and reviewing the GRPO algorithm proposed by DeepSeek. 2.1 Reinforcement Learning Setup Let s_{0} denote the initial prompt. At each time step t , the language model generates a token a_{t}\in\mathbb{R}^{d} (a column vector), forming a sequence of at most T tokens. We define the state at time t as s_{t}:=(s_{0},a_{1},\dots,a_{t})^{\top}\in\mathbb{R}^{t\times d}. To ensure consistent dimensionality across time steps, we embed each state into a fixed-dimensional space \mathbb{R}^{T\times d} via zero-padding: s_{t}:=(s_{0},a_{1},\dots,a_{t},\underbrace{0,\dots,0}_{(T-t)\ \text{tokens}})^{\top}, where the final T-t entries are zero vectors in \mathbb{R}^{d} . We assume the existence of a predefined reward function r(s):\mathbb{R}^{T\times d}\to\mathbb{R} , which evaluates the quality of a complete generated sequence s . The LLM is modeled as a parameterized policy. Let \displaystyle\pi_{\theta}(a\mid s):\mathbb{R}^{T\times d}\rightarrow[0,1] (1) denote the probability of generating token a given current state s , under model parameters \theta\in\mathbb{R}^{T\times d} . This policy corresponds to the LLM’s output distribution conditioned on its current context. We adopt the standard RL formulation: the state space S consists of token sequences so far, and the action space A corresponds to the vocabulary. In many LLM tasks, rewards are sparse and provided only at the final step (i.e., at t=T ), which aligns with the structure of post-hoc evaluation models used in practice. We now define the trajectory probability and value function. The joint probability of generating a full trajectory under policy \pi_{\theta} is given by: \Pro_{\theta}(s_{T}\mid s_{0})=\prod_{t=1}^{T}\pi_{\theta}(s_{t}\mid s_{t-1}). The goal is to maximize the expected return: \displaystyle J(\theta)=\mathbb{E}_{s_{T}\sim\pi_{\theta}}\left[r(s_{T})\right]=\sum_{s_{T}\in\mathcal{S}_{T}}\Pro_{\theta}(s_{T}|s_{0})r(s_{T}), (2) where \mathcal{S}_{T} denotes the space of all trajectories of length T . To encourage the new policy to stay close to a reference policy \pi_{\text{ref}} , a KL-penalized objective is often used: \displaystyle\mathcal{J}(\theta):=J(\theta)-\beta\mathrm{KL}\bigl(\pi_{\theta}\|\pi_{\theta_{\text{ref}}}\bigr), (3) where \beta\geq 0 is a regularization coefficient. The optimization of \mathcal{J}(\theta) typically follows a gradient ascent (GA) scheme: 1 1 1 We use gradient ascent as the goal is to maximize \mathcal{J}(\theta) . Gradient descent is equivalent up to a sign change. \theta_{n+1}=\theta_{n}+\eta\nabla_{\theta_{n}}\mathcal{J}(\theta_{n}),

Read the complete text