Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang · 2026

Read Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients by Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23135.txt.

Introduction

Cheng Ge, Caitlyn Heqi Yin, Hao Liang, and Jiawei Zhang explain why a popular training method needs its normalization step. Group Relative Policy Optimization, or GRPO, trains language models without a separate critic network. It compares answers within a group per prompt and normalizes by variance.

Why that normalization helps, and when, was unclear. The authors give a local-curvature view of the adaptive gradients involved. A clear reason for a standard trick helps practitioners tune it instead of trusting habit. Per-prompt baselines replace the critic, so understanding the normalization explains the method's core stability trick.

Catalog record

CREATORCheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-23135

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang Abstract Reinforcement learning (RL) has become a key driver of language model reasoning. Among RL algorithms, Group Relative Policy Optimization (GRPO) is the de facto standard, avoiding the need for a critic by using per-prompt baselines and variance normalization. Yet why and when this normalization helps remains unclear. In this work, we provide an explanation through the lens of local curvature of the sequence-level policy gradient: standard deviation normalization implements an adaptive gradient. Theoretically, under mild conditions, GRPO enjoys a strictly improved convergence rate over unnormalized REINFORCE, with gains characterized by the average within-prompt reward standard deviation across prompts and iterations. Empirically, our analysis on GSM8K and MATH benchmarks reveals three distinct training phases governed by the interplay between feature orthogonality and reward variance: (I) an early acceleration phase where high variance and orthogonality favor adaptive scaling; (II) a relatively stable transition phase; and (III) a late-stage regime where the loss of orthogonality limits further gains. Together, these results provide a principled account of when std normalization helps in GRPO, and offer broader insights into the design of critic-free RL algorithms. † † affiliationnotice: Equal contribution 1 Introduction Large language models (LLMs) have recently exhibited striking gains in multi-step reasoning, particularly when a lightweight reinforcement-learning (RL) stage is applied on top of a strong and pretrained base model. Among the many post-training recipes, Group Relative Policy Optimization (GRPO) has emerged as a practical, critic-free alternative that has powered some of the most visible reasoning systems ( 30 ) , where it consistently improves solution accuracy under tight compute budgets. While Proximal Policy Optimization (PPO) ( 29 ) remains a popular default in RLHF pipelines, it couples the policy with a learned value-function critic (and often GAE ( 28 ) ). This increases memory footprint and implementation complexity, since the environment is a pretrained LLM and rewards arrive only at the end of whole sequences. This has renewed interest in critic-free policy-gradient methods that operate at the sequence level, such as REINFORCE-style method, including ReMax ( 17 ) , and RLOO ( 2 ) , which often match or outperform PPO for LLM alignment while being simpler and lighter-weight. Classical REINFORCE ( 36 ) reduces gradient variance by subtracting a baseline (e.g., a running mean reward), yielding an unbiased estimator with lower variance ( 9 ) . This is textbook policy-gradient variance reduction. GRPO goes one step further: for each prompt, it samples multiple responses from the current policy, computes the group mean reward as a baseline, and normalizes each response’s update by the within-prompt reward standard deviation, which effectively uses a per-prompt z-scored advantage ( 30 ) . Empirically, this simple normalization has been repeatedly observed to stabilize optimization and improve sample-efficiency in LLM RL ( 16 ) . Yet, the underlying mechanism of this normalization step has not been theoretically clarified. What, exactly, does normalization do? In this work, we provide a principled explanation of why and when GRPO benefits from it. Our key insight is that the reward variance of each question serves as an estimate of the local Lipschitz constant (curvature) of the policy gradient. Standard-deviation normalization therefore implements an adaptive gradient rule, scaling each update by an inverse-curvature proxy and effectively assigning a larger step size to smoother prompts and a smaller step size to sharper prompts. This mechanism improves stability and convergence precisely when (i) curvature is heterogeneous across prompts and evolves over training, and (ii) cross-prompt interference is controlled so that prompt-wise adaptive steps are not overwhelmed by interaction noise. This perspective explains why GRPO can outperform unnormalized policy gradient methods in practice. We characterize concrete conditions under which normalization yields gains both theoretically and empirically. To summarize, our contributions are: • Curvature interpretation. We provide a new perspective on GRPO as a principled adaptive gradient mechanism that adapts learning to per-prompt curvature. • When normalization provably helps. We show that under reasonable and mild assumptions, GRPO attains a faster convergence rate than unnormalized REINFORCE, with the speedup explicitly governed by the average per-prompt standard deviation factors over time. • When normalization empirically helps and when it saturates . On GSM8K and MATH, we validate the curvature–variance link and uncover three regimes for normalization: (I) early, near-orthogonal high-variance training where gains can be muted by noisy gradients; (II) a mid-stage with moderate variance and stable interactions where normalization delivers the strongest and most consistent improvements, especially on harder prompts; and (III) a late stage where reduced orthogonality increases interference and the marginal benefit plateaus. Our results explain why GRPO needs normalization: it is a principled adaptive gradient mechanism whose benefits emerge under identifiable geometric and statistical conditions. This provides a theoretical foundation for the empirical success of GRPO and offers broader insights into the design of critic-free RL algorithms for LLM training. 1.1 Related works REINFORCE-style PG methods (prior to GRPO). 17 proposes a sequence-level REINFORCE-style algorithm, ReMax, for LLM alignment. 2 proposes RLOO, which samples multiple responses per prompt and uses a leave-one-out baseline to further reduce variance. GRPO and its variants. GRPO ( 30 ) and related methods ( 31 ; 21 ; 5 ; 19 ) have gained widespread adoption due to their simplicity and scalability, and are now commonly used in post-training pipelines for reasoning-oriented models. Several extensions improve stability in long-CoT RL. DAPO ( 38 ) introduces several techniques, including higher clipping thresholds and sequence-level losses, to make the training more stable. GSPO ( 41 ) replaces the sequence-level importance weight with sequence likelihood to avoid high-variance noise. CISPO ( 4 ) further revises the clipping strategy to avoid low-probability tokens from being clipped out after the first on-policy update. Emerging theory for GRPO. Recent studies analyze what GRPO optimizes and how it behaves in on- and off-policy regimes ( 24 ) , its implicit alignment objective ( 34 ) , and trajectory-corrected variants with convergence guarantees ( 26 ) . 3 highlights a trade-off between normalization and calibration, showing that removing the std term can improve probability calibration at the cost of optimization speed. 14 interprets GRPO’s optimization as a weighted combination of maximization likelihood for correct rollouts and minimization for the incorrect ones, while noting that the use of std normalization lacks a clear theoretical motivation. We contribute a new perspective, interpreting the std term as an adaptive gradient mechanism reflecting local curvature, which helps unify a range of previously disparate empirical findings. RLVR. Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective paradigm for reasoning-intensive domains. Unlike RLHF, which relies on a learned reward model, RLVR uses deterministic, verifiable rewards ( 15 ; 10 ; 33 ; 35 ) . This mitigates reward-model bias and hacking and simplifies training, while scaling effectively with compute and dataset size. Strong results have been reported on the well-recognized GSM8K ( 6 ) , MATH ( 11 ) , Omni-MATH ( 7 ) , and FormalMATH ( 39 ) . In this paper, we focus on developing a theoretical understanding of GRPO in the RLVR setting. 2 Preliminaries and Problem Settings We begin by introducing the RLVR framework for

Read the complete text