MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
Youngeun Kim * · 2026
Read MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning by Youngeun Kim * (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22582.txt.
Introduction
Youngeun Kim strengthens a group-based training method for the case of few tries per prompt. Group-relative methods grade a batch of model replies against their shared mean. Their 2026 preprint, MC-GRPO, notes that with small batches that mean is noisy.
Noise can flip the sign of the learning signal, teaching the wrong lesson. Centering on the median, not the mean, steadies the baseline. Steadier signals matter most where compute is tight. The fix is small but pointed. It aims to keep cheap training honest. Small budgets are where most teams live.
Catalog record
| CREATOR | Youngeun Kim * |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22582
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning Youngeun Kim * Abstract Group-relative policy optimization methods train language models by generating multiple rollouts per prompt and normalizing rewards with a shared mean reward baseline. In resource-constrained settings where the rollout budget is small, accuracy often degrades. We find that noise in the shared baseline induces advantage sign flips, where some rollouts receive an incorrect advantage sign, and the update direction is reversed. To address this, we propose Median-Centered Group Relative Policy Optimization (MC-GRPO), a simple and effective solution for small-rollout training. Our main idea is to replace the mean baseline with a median baseline: the median is far less sensitive to outlier rewards than the mean, mitigating the sign flips under small rollout size ( G G ). We generate one additional rollout for median reference ( G + 1 G+1 ), and compute advantages by using the group median. With an odd-sized group, exactly one completion is the median and receives zero advantage, we exclude this pivot rollout from backpropagation so the number of gradient-contributing samples per prompt remains G G , preserving the core update cost of standard G G -rollout training. Across various GRPO-family methods and a wide range of models and scales, this median-centered training consistently improves stability and final accuracy in the low-rollout regime, reducing the gap between G = 2 G{=}2 and G = 8 G{=}8 to within 1 % 1\% . Code is available at lotusroot-kim/MC-GRPO . † † affiliationnotice: * Most of this work was completed prior to joining Amazon. 1 Introduction Online reinforcement learning (RL) for large language models (LLMs) has recently emerged as an effective paradigm for directly optimizing task objectives from rule-based rewards, learned reward models, or preference signals ( 31 ; 3 ; 24 ; 1 ; 7 ; 36 ) . A common practice in this setting is to sample multiple responses ( i.e. , rollouts) per prompt and learn from their relative quality. Group-relative methods such as GRPO ( 32 ) instantiate this idea by computing advantages within each prompt group using a shared baseline. Given rewards \{r_{i}\}_{i=1}^{G} for G rollouts, GRPO subtracts the within-prompt mean reward to produce a centered, relative learning signal that has been shown to improve training stability and downstream accuracy. Figure 1 : Accuracy (%) versus the number of rollouts for Qwen3-1.7B trained on GSM8K. We compare the original GRPO, DAPO, and DR-GRPO methods ( ; baselines) with their Median-Centered (MC) variants ( ; ours). MC training improves robustness and yields larger gains under small rollout budgets (2 \sim 4 rollouts), while remaining competitive at higher rollout counts. Despite its effectiveness at high rollout counts ( e.g. , 8–32), this design becomes fragile when training is resource-constrained. In practice, the number of rollouts per prompt is often limited by throughput, latency, or memory constraints ( 37 ; 16 ) , especially for individual researchers or academic settings with limited GPU resources. From a standard RL perspective, reducing the rollout budget naturally hurts performance (Fig. 1 ) because it weakens exploration: with fewer rollouts, the policy observes fewer diverse candidates and is less likely to sample rare but high-quality trajectories ( 35 ) . In our work, we find that reduced exploration is not the only story behind performance degradation in the small-rollout regime. In addition, accuracy drops are driven by advantage sign flips (Sec. 2 ). Specifically, in GRPO-style methods, the advantage for each rollout is computed relative to the mean reward across rollouts, so higher-reward completions should receive positive advantages while lower-reward ones receive negative advantages (Eq. 1 ). When G is small, a single outlier reward can substantially shift this mean baseline, causing multiple rollouts to flip the sign of their advantages. Such sign flips reverse the intended update direction, i.e. , reinforcing worse trajectories and penalizing better ones; this effect compounds over training and leads to lower final performance (even a 5% sign-flip rate induces a \sim 4% accuracy drop). To address this issue, we propose Median-Centered Group Relative Policy Optimization (MC-GRPO), replaces mean-centering with median-centering when computing the shared within-prompt baseline. The median is a classical robust estimator that is far less sensitive to outliers, and in our setting it directly targets the baseline-induced advantage sign-flip failure mode. Concretely, we sample one extra rollout per prompt to form an odd-sized group (G{+}1) . With an odd-sized group, exactly one completion is the median and therefore receives zero advantage; we exploit this property and exclude the median sample from backpropagation, keeping the number of gradient-contributing samples per prompt equal to G and thus preserving the core update cost of the standard G -rollout method. In practice, only one additional rollout often brings negligible cost with high-throughput inference engines ( e.g. , vLLM). Importantly, MC training is algorithm- and model-agnostic within the GRPO family: applying the same median-centered baseline yields consistent improvements in both training stability and final task performance across GRPO, DAPO ( 40 ) , and DR-GRPO ( 17 ) . We also observe the same trend across a broad set of models ranging from small to mid-sized LLMs, including Llama-3.2-3B-Instruct ( 5 ) , Qwen2.5-Math-1.5B ( 27 ) , Qwen3-1.7B, Qwen3-4B-Instruct ( 28 ) , and Qwen2.5-7B-Instruct ( 26 ) . Our proposed method substantially narrows the performance gap between low- and high-rollout regimes. For Qwen3-1.7B on GSM8K, GRPO achieves 78.90% with 2 rollouts and 84.53% with 8 rollouts. With our method, the 2-rollout setting improves to 83.54%, substantially narrowing the gap to the 8-rollout result. Our contribution can be summarized as follows: (1) We identify baseline-induced advantage sign flips as a key driver of the accuracy drop in GRPO-style online RL under small rollout budgets. When the shared mean baseline is noisy, reversing update directions and compounding errors over training. (2) We propose MC-GRPO, a simple and effective method that samples one additional rollout to form an odd-sized group and replaces mean-centering with robust median-centering. The median sample has zero advantage and can be excluded from backprop, preserving the core update cost while directly targeting sign-flip instability. (3) Across various GRPO-family algorithms and across diverse model families and scales, MC training consistently improves stability and final accuracy in the low-budget regime, achieving \leq 1\% gap between 2 -rollout and 8 -rollout. 2 Motivation Figure 2 : Sign flips are frequent under small rollout budgets. (a) With few rollouts, the sample-mean baseline can shift substantially depending on which rollouts are included, causing an advantage sign flip for the same trajectory ( e.g. , the 0.5 -reward sample flips sign when the rollout set changes from 8 to 4 ). We also report the sign-flip rate ( i.e. the fraction of rollouts whose advantage sign under a k -rollout baseline disagrees with an oracle sign computed from G_{\mathrm{ref}}{=}128 rollouts) for (b) Qwen3-1.7B/GSM8K, (c) Qwen2.5-7B-Instruct/Math-500, and (d) Llama-3.2-3B-instruct/Math-500. For each setting, we evaluate 250 prompts; for each prompt and k\in\{2,4,8\} , we draw 20 random k -subsamples from the 128 rollouts, compute either the mean or median baseline, and average the resulting sign-flip rates. Group-Relative Policy Optimization (GRPO) and its variants adapt PPO by computing advantages relative to other rollouts from the same prompt ( 32