Transform-Augmented GRPO Improves Pass@k
Khiem Le, Youssef Mroueh, Phuc Nguyen, Chi-Heng Lin, Shangqian Gao, Ting Hua, Nitesh V. Chawla · 2026
Read Transform-Augmented GRPO Improves Pass@k by Khiem Le, Youssef Mroueh, Phuc Nguyen, Chi-Heng Lin, Shangqian Gao, Ting Hua, Nitesh V. Chawla (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22478.txt.
Introduction
Khiem Le and six co-authors lift a training method by showing it the same task in fresh dress. Their 2026 preprint notes that models learn by next-word prediction, which makes them pattern matchers at heart. A reworded task can trip them, though the core ask is the same.
Their tweak, Transform-Augmented GRPO, trains on such reworded forms. The aim is to lift Pass@k, the chance that one of a few tries is right. Robustness to wording is key for real use, where no two users ask alike. The work makes training match that fact.
Catalog record
| CREATOR | Khiem Le, Youssef Mroueh, Phuc Nguyen, Chi-Heng Lin, Shangqian Gao, Ting Hua, Nitesh V. Chawla |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22478
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transform-Augmented GRPO Improves Pass@k Khiem Le, Youssef Mroueh, Phuc Nguyen, Chi-Heng Lin, Shangqian Gao, Ting Hua, Nitesh V. Chawla Abstract Large language models trained via next-token prediction are fundamentally pattern-matchers: sensitive to superficial phrasing variations even when the underlying problem is identical. Group Relative Policy Optimization (GRPO) was designed to improve reasoning, but in fact it worsens this situation through two failure modes: diversity collapse , where training amplifies a single solution strategy while ignoring alternatives of gradient signal, and gradient diminishing , where a large portion of questions yield zero gradients because all rollouts receive identical rewards. We propose TA-GRPO (Transform-Augmented GRPO), which generates semantically equivalent transformed variants of each question (via paraphrasing, variable renaming, and format changes) and computes advantages by pooling rewards across the entire group. This pooled computation ensures mixed rewards even when the original question is too easy or too hard, while training on diverse phrasings promotes multiple solution strategies. We provide theoretical justification showing that TA-GRPO reduces zero-gradient probability and improves generalization via reduced train-test distribution shift. Experiments on mathematical reasoning benchmarks show consistent Pass@k improvements, with gains up to 9.84 points on competition math (AMC12, AIME24) and 5.05 points on out-of-distribution scientific reasoning (GPQA-Diamond). 1 Introduction What Pretraining Creates: A Pattern-Matcher. Large Language Models (LLMs) are trained via next-token prediction: given a context, predict the probability distribution over the next token. This objective teaches the model which tokens tend to appear in which contexts. Fundamentally, it creates a sophisticated pattern-matcher rather than a system that truly “understands” language or reasoning. This pattern-matching nature explains a well-known phenomenon: LLMs are sensitive to superficial variations in input phrasing. Small changes in wording ( Salinas & Morstatter 2024 ) , question format ( Hua et al. 2025 ) , variable names, or even the order of information ( Guan et al. 2025 ) can dramatically change model outputs, even when the underlying problem is semantically identical. A model that truly understood would be invariant to such surface-level changes; a pattern-matcher is not. What GRPO Is Supposed To Do. To bridge the gap between token prediction and problem-solving, researchers turn to Reinforcement Learning (RL): optimize directly for task success rather than next-token likelihood. For reasoning tasks like mathematics or coding, rewards can be verified automatically: the answer is either correct or not. This setting is known as Reinforcement Learning with Verifiable Rewards (RLVR) ( Lambert et al. 2025 ) . Among RLVR algorithms, Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) is the most widely used. While traditional policy gradient methods require a separate value function, GRPO computes advantages by comparing rollouts within each question group. The success of DeepSeek-R1 ( Guo et al. 2025 ) demonstrates that this simpler approach is effective, and GRPO has since been widely adopted for training reasoning models. The hope is that GRPO will teach the model to “reason better”, to develop robust problem-solving strategies that generalize beyond the training distribution. What GRPO Actually Does: Making Pattern-Matching Worse. In practice, GRPO does not fix the pattern-matching limitation of pretrained models; it worsens it. (1) Diversity Collapse. GRPO amplifies whatever solution pattern currently achieves the highest reward. If the model has a dominant strategy for a question (e.g., solving algebraically rather than geometrically), GRPO reinforces this pattern while starving alternative valid patterns of gradient signal. The result is a model that collapses from knowing multiple solution strategies to knowing only one ( Yue et al. 2025 ; Zhang et al. 2025 ; Dang et al. 2025 ) . This manifests as suboptimal Pass@k performance, the probability of finding at least one correct answer in k attempts. Because the model repeatedly produces the same answer rather than exploring diverse solutions. (2) Gradient Diminishing. GRPO’s group-relative normalization creates a problematic case: when all rollouts for a question receive identical rewards (all correct or all incorrect), the computed advantages are zero, yielding no gradient signal. Questions that are “too easy” (the model always succeeds) or “too hard” (the model always fails) contribute nothing to learning. As shown in Figure 1 , over 80% of training questions yield zero gradients for standard GRPO, with only medium-difficulty questions contributing to learning. Figure 1 : Percentage of zero-gradient questions throughout training (Qwen3-1.7B). Questions that are “too easy” (all rollouts correct) or “too hard” (all rollouts incorrect) yield zero gradients and contribute nothing to learning: lower is better. TA-GRPO consistently reduces zero-gradient questions compared to GRPO, maintaining a 12–16 percentage point advantage throughout training (e.g., 75% vs 86% at convergence). Our Insight: Diversify the Training Signal. The root cause of both failures is that GRPO trains on each question in only one phrasing, reinforcing a single dominant pattern. Our insight is simple: present each question in multiple semantically equivalent forms. This forces the model to encounter diverse patterns for the same underlying problem, breaking the single-pattern trap. We propose transform augmentation : training on multiple phrasings of each question. By presenting “What is 2+3 ?”, “Calculate 2+3 ”, “Find the sum of 2 and 3”, and “If x=2+3 , what is x ?” as variations of the same problem, we expose the model to diverse solution patterns rather than reinforcing just one. Transform augmentation directly addresses these issues. First, different phrasings trigger different dominant patterns, so training on diverse phrasings ensures multiple solution strategies receive gradient signal. Second, different phrasings have different difficulty, so grouping a question with its transforms produces non-zero advantages even for otherwise “too easy” or “too hard” questions. Third, training on diverse phrasings reduces the train-test distribution shift. Our main contributions are as follows: • TA-GRPO Algorithm ( Section 4 ): We propose TA-GRPO (Transform-Augmented GRPO), which groups each question with its transformed variants for joint advantage computation, directly addressing gradient diminishing while promoting diversity. • Theoretical Framework ( Section 5 ): We provide theoretical justification through four mechanisms: (1) non-vacuous initialization: it suffices that any transform succeeds; (2) reduced gradient diminishing: we prove TA-GRPO has lower zero-gradient probability than standard GRPO; (3) a Pinsker-KL generalization bound showing the train-test gap is controlled by KL divergence; and (4) a principled derivation of the pooled advantage formula from Bernoulli variance. • Experimental Validation ( Section 6 ): We demonstrate consistent Pass@k improvements on mathematical reasoning benchmarks. On competition-level problems (AMC12 and AIME24), TA-GRPO improves Pass@32 by up to 9.84 points. On out-of-distribution scientific reasoning (GPQA-Diamond), the improvement reaches 5.05 points, confirming the generalization benefits of transform augmentation. 2 Related Work GRPO and Variants. GRPO ( Shao et al. 2024 ) has become the dominant method for training reasoning models due to its simplicity and efficiency. However, its limitations have spurred numerous modifications. DAPO ( Yu et al. 2025 ) addresses gradient diminishing through dynamic sampling, over-sampling rollouts and filtering out questions where all rollouts receive identical rewards. While effective, this approach discards potentially valuable training signal. DCPO ( Yang et al. 2025b ) extends this with adaptive clipping bounds. Dr. GRPO ( Liu et al. 2025 ) removes the standard deviation normalization entirely to avoid difficulty-dependent biases, while GRPO-LEAD ( Zhang & Zuo 2025 ) employs difficulty-aware weighting to rebalance the contribution of easy versus hard questions. Other variants address different aspects: GRPO-MA