Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
Bryan L. M. de Oliveira, Felipe V. Frujeri, Marcos P. C. M. Queiroz, Luana G. B. Martins, Telma W. de L. Soares, Luckeciano C. Melo · 2025
Read Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments by Bryan L. M. de Oliveira, Felipe V. Frujeri, Marcos P. C. M. Queiroz, Luana G. B. Martins, Telma W. de L. Soares, Luckeciano C. Melo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03527.txt.
Introduction
Bryan L M de Oliveira and co-authors run the first systematic check of a critic-free learning method in classic control tasks. Group-relative policy optimization drops the learned critic used by its famous predecessor and grades whole trajectory groups against each other instead. That cut raises a basic question: are learned baselines needed at all?
Careful ablations isolate baselines, discounting, and group sampling across discrete and continuous tasks. Learned critics prove essential for long-horizon work, where every critic-free variant trails the classic method. Short tasks tell another story, with simple episodic grading holding its own. The findings map where the cheaper method is safe to use and where the critic earns its keep.
Catalog record
| CREATOR | Bryan L. M. de Oliveira, Felipe V. Frujeri, Marcos P. C. M. Queiroz, Luana G. B. Martins, Telma W. de L. Soares, Luckeciano C. Melo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-03527
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments Bryan L. M. de Oliveira, Felipe V. Frujeri, Marcos P. C. M. Queiroz, Luana G. B. Martins, Telma W. de L. Soares, Luckeciano C. Melo Abstract Group Relative Policy Optimization (GRPO) has emerged as a scalable alternative to Proximal Policy Optimization (PPO) by eliminating the learned critic and instead estimating advantages through group-relative comparisons of trajectories. This simplification raises fundamental questions about the necessity of learned baselines in policy-gradient methods. We present the first systematic study of GRPO in classical single-task reinforcement learning environments, spanning discrete and continuous control tasks. Through controlled ablations isolating baselines, discounting, and group sampling, we reveal three key findings: (1) learned critics remain essential for long-horizon tasks—all critic-free baselines underperform PPO except in short-horizon environments like CartPole where episodic returns can be effective; (2) GRPO benefits from high discount factors ( γ = 0.99 \gamma=0.99 ) except in HalfCheetah, where lack of early termination favors moderate discounting ( γ = 0.9 \gamma=0.9 ); (3) smaller group sizes outperform larger ones, suggesting limitations in batch-based grouping strategies that mix unrelated episodes. These results reveal both the limitations of critic-free methods in classical control and the specific conditions where they remain viable alternatives to learned value functions. 1 Introduction Policy gradient methods form the backbone of modern reinforcement learning, directly optimizing policies to maximize expected rewards 25 ; 21 . Among these, Proximal Policy Optimization (PPO) 18 has emerged as a dominant approach, combining stability with sample efficiency through its use of a learned value function (critic) and clipped objective. However, the computational overhead of maintaining both actor and critic networks has motivated the development of critic-free alternatives, particularly in resource-constrained settings such as large language model (LLM) training. Group Relative Policy Optimization (GRPO) represents one such alternative, originally developed for reinforcement learning from human feedback (RLHF) in LLMs 19 . GRPO eliminates the critic entirely, instead estimating advantages by comparing returns across groups of trajectories sampled from the same state or prompt. While this approach has shown promise in language model fine-tuning, its behavior in classical reinforcement learning environments remains largely unexplored. The theoretical foundation for baseline methods in policy gradients is well-established. REINFORCE 25 introduced the use of baselines to reduce variance without introducing bias, where the gradient estimator subtracts a baseline b(s) from the return R_{t} . Common baseline choices include scalar averages, learned state-value functions, or advantage functions. The latter approach led to the development of actor-critic methods like A2C and A3C 12 . PPO builds upon this foundation by using a learned critic to estimate advantages, typically combined with Generalized Advantage Estimation (GAE) for bias-variance control. Its clipped objective prevents destructively large policy updates while maintaining sample efficiency. GRPO replaces the learned critic with a group-based baseline computed from episode returns. Given a group of G sampled episodes from the same initial state, GRPO normalizes the returns across the group to estimate advantages. This group-relative normalization removes the need for a learned value function while providing a baseline that adapts to the current policy’s performance distribution. This work addresses a fundamental question in policy gradient methods: How does GRPO compare against PPO and its variants in standard RL benchmarks? We conduct the first systematic evaluation of GRPO across classical reinforcement learning environments, spanning discrete control (CartPole, Acrobot), and continuous control (MountainCarContinuous, HalfCheetah, Humanoid). Our study is designed around controlled ablations that examine three key differences: (1) baseline estimation, comparing GRPO’s group-relative normalization against PPO’s learned critic and simpler alternatives; (2) horizon and discounting, analyzing GRPO’s preference for \gamma=1 across different task structures; and (3) grouping strategies, evaluating how group sizes affect performance. Our findings reveal that GRPO’s effectiveness is highly context-dependent in classical RL settings. We show that critic-free baselines cannot replace learned value functions in long-horizon continuous control tasks—GRPO performs comparably to simple batch baselines but substantially underperforms PPO across all environments except CartPole. The presence of early termination emerges as a critical factor determining when critic-free methods can extract meaningful learning signals. GRPO generally benefits from high discount factors ( \gamma=0.99 often optimal), with notable exceptions in environments lacking early termination like HalfCheetah (which favors \gamma=0.9 ). Smaller group sizes surprisingly outperform larger ones, even when controlling for update frequency, suggesting limitations in batch-based grouping strategies that mix potentially unrelated episodes from different parallel environments. These results establish the first systematic evaluation of GRPO beyond language modeling, revealing both its limitations and the conditions under which critic-free methods remain viable. To facilitate reproducibility, we provide code, configuration files, and experiment logs that reproduce all results reported in this paper at https://github.com/AKCIT-RL/revisiting-grpo . 2 Related Work Policy Gradient Methods and Baselines. Policy gradient methods originated with REINFORCE 25 , with the policy gradient theorem formalized for function approximation 21 . Because Monte Carlo returns have high variance, baselines act as control variates to reduce variance without bias 25 ; 5 . The optimal state-dependent baseline minimizing variance was shown to be the value function itself 24 . Recent work also revisits the original REINFORCE algorithm proposing alternative gradient estimation techniques 3 . Actor-critic methods (e.g., A2C/A3C 12 , NAC 14 ) learn V(s) to produce state-dependent advantages; GAE further improves bias–variance trade-offs 17 . Beyond value baselines, control-variates such as Q-Prop and action-dependent baselines also target variance in continuous control 6 ; 26 , though more complex action-dependent baselines do not necessarily reduce variance over standard state-dependent baselines in practice 23 . To stabilize updates, trust-region and natural-gradient methods 10 culminate in TRPO 16 , with PPO’s clipped surrogate providing a first-order approximation to conservative updates 9 ; 18 . RL in Large Language Models. With the advancement of Large Language Models (LLMs), the challenge of aligning their results with human intentions arises. RLHF fine-tunes pretrained LMs using KL-regularized PPO-like objectives 4 ; 28 ; 20 ; 13 . PPO is typically adapted to start from a supervised fine-tuned (SFT) reference policy with a KL control term to regularize updates, stabilizing optimization in high-dimensional action spaces. Due to its robustness, PPO quickly became the default algorithm for the optimization step in the RLHF pipeline 29 . In the case of language models, this burden is amplified, as PPO training typically requires four models in memory: the actor (policy), the critic (value function), a frozen reference policy for KL regularization, and the reward model, although in practice the actor and critic may share parameters to reduce overhead. It was in this context that Group Relative Policy Optimization (GRPO) 19 was introduced, proposing a simpler approach. Several alternative approaches address the computational and implementation challenges of PPO in LLMs. Direct Preference Optimization (DPO) 15 and Sequence Likelihood Calibration (SLiC-HF) 27 reformulate alignment as contrastive learning problems, training the model to assign higher likelihood to preferred sequences over dispreferred ones without the need for rollouts against a critic or a frozen reference policy. Reinforcement Learning from AI Feedback (RLAIF) 2 extends the alignment paradigm by replacing human-labeled preference data with synthetic feedback generated by large language models themselves, further reducing the cost and memory requirements of training. Collectively, these methods share the goal of simplifying the optimization pipeline, reducing reliance on multiple large neural