Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets
Benjamin Pikus Pratyush Ranjan Tiwari Burton Ye, [0.5em] Independent Eternis Labs Writer, Inc · 2025
Read Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets by Benjamin Pikus Pratyush Ranjan Tiwari Burton Ye, [0.5em] Independent Eternis Labs Writer, Inc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14094.txt.
Introduction
Benjamin Pikus, Pratyush Tiwari, and Burton Ye showed in 2025 that hard practice problems, not easy ones, make reasoning models improve on a budget. Labeling training data costs money, so which examples to buy matters. Using a group-based reinforcement method, they trained models on easy, medium, hard, or random slices of the same pools.
The hardest tenth, where the base model mostly failed, delivered gains up to forty-seven percent, while easy slices barely moved scores. Mixed wins and losses keep the learning signal alive. Hard-trained models also generalized best to a fresh contest benchmark.
Catalog record
| CREATOR | Benjamin Pikus Pratyush Ranjan Tiwari Burton Ye, [0.5em] Independent Eternis Labs Writer, Inc |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-14094
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets Benjamin Pikus Pratyush Ranjan Tiwari Burton Ye, [0.5em] Independent Eternis Labs Writer, Inc Abstract Collecting high-quality training examples for language model fine-tuning is expensive, with practical budgets limiting the amount of data that can be procured. We investigate a critical question for resource-constrained alignment: under a fixed acquisition budget, should practitioners prioritize examples that are easy, medium, hard, or of random difficulty? We study Group Relative Policy Optimization (GRPO) fine-tuning across different model sizes and families, comparing four subset selection policies chosen from the same unlabeled pool using base-model difficulty estimates obtained via multi-sample evaluation. Our experiments 2 2 2 Code available at https://github.com/Pikus16/grpo_difficulty reveal that training on the hardest examples yields the largest performance gains, up to 47%, while training on easy examples yield the smallest gains. Analysis reveals that this effect arises from harder examples providing more learnable opportunities during GRPO training. These findings provide practical guidance for budget-constrained post-training: prioritizing hard examples yields substantial performance gains on reasoning tasks when using GRPO. 1 Introduction Large language model (LLM) alignment and post-training are fundamentally constrained by the expense of acquiring high-quality supervision data, making the selection of training examples a critical factor in achieving optimal model performance [ 17 ] . While recent advances in reinforcement learning from human feedback (RLHF) and related techniques have demonstrated remarkable improvements in model capabilities, practitioners face a practical challenge: given limited resources for data annotation and curation, which examples should be prioritized to maximize post-training performance? In this work, we focus on Group Relative Policy Optimization (GRPO), a PPO‑style algorithm that replaces a learned value function with group‑normalized advantages, reducing memory and relying on within‑group reward variance for learning signals. We address a specific instantiation of this budget-aware selection problem: given a fixed budget to select and train on only a fraction of available prompts, how should we choose this subset to maximize the effectiveness of Group Relative Policy Optimization (GRPO) fine-tuning? Specifically, we investigate whether selecting examples based on their difficulty (as measured by the base model’s success rate across multiple sampling attempts) leads to systematic differences in final model performance on held-out test sets. Our key research questions are: 1. Should practitioners prioritize hard examples (where the base model frequently fails), easy examples (where it frequently succeeds), middle-difficulty examples, or a random selection? 2. Does the optimal selection strategy vary across model scales and families? 3. What mechanisms explain the differential effectiveness of difficulty-based selection strategies under GRPO’s learning dynamics? 4. How do different difficulty-based selection strategies impact out-of-distribution generalization to substantially harder problem sets? To answer these questions, we make the following contributions: Budget-aware evaluation protocol. We develop a systematic framework for comparing difficulty-conditioned training subsets under GRPO, using multi-sample base-model probing to robustly estimate example difficulty [ 26 ] . This protocol enables fair comparison across selection strategies while controlling for computational budget and other confounding factors. Comprehensive experimental evaluation. We conduct extensive experiments on GSM8K grade-school math problems and BIG-Bench Hard’s Tracking Shuffled Objects task across models (Qwen3-4B, Qwen3-14B, Phi-4, and Llama3.1-8B) [ 6 , 29 , 1 ] . Our results consistently show that training on the hardest 10% of examples yields superior test performance on reasoning tasks. Mechanistic understanding of selection effects. We analyze why hard examples prove most effective under GRPO: the algorithm requires within-group outcome variance to generate learning signals. When all samples in a group produce identical rewards, the advantages become zero and learning stops. Our analysis shows that hard examples maintain mixed outcomes longer than easy examples, which quickly converge to deterministic success [ 20 ] . Out-of-distribution generalization analysis. We evaluate models trained on different difficulty-based subsets against the AIME2025-I benchmark, a substantially harder test set than the in-distribution GSM8K [ 16 , 2 ] . We find that training on the hardest examples not only improves in-distribution performance, but is also the only strategy that yields meaningful gains on OOD data. Our findings have immediate practical implications: when using GRPO for reasoning task fine-tuning under budget constraints, practitioners should prioritize collecting and annotating examples where the base model struggles but can occasionally succeed. This hard-example focus can yield performance improvements exceeding 30 percentage points over easy selection on some configurations, transforming marginally effective fine-tuning into highly successful model improvement. Our study isolates offline difficulty‑based selection under GRPO. It complements online difficulty‑targeted selection with rollout replay that prioritizes moderate difficulty [ 21 ] , advances in process‑reward RL (PRIME) [ 5 ] , and analyses of GRPO optimization biases such as Dr. GRPO [ 15 ] . We focus on outcome‑reward GRPO; extending our protocol to dense process rewards or debiased GRPO is left to future work. The remainder of this paper is organized as follows. Section 2 presents our experimental protocol for difficulty estimation and subset selection. Section 3 reports our main experimental results across tasks and model scales. Section 4 provides detailed analysis of learning dynamics and model behavior. Section 5 concludes with practical recommendations and future directions. Related work can be found in the appendix. 2 Experimental Protocol This section describes our experimental framework for evaluating the impact of example difficulty on GRPO fine-tuning under budget constraints. We detail our difficulty estimation procedure, subset selection policies, training methodology, and evaluation metrics. 2.1 Goal Our primary research goal is to determine which subset selection strategy maximizes post-GRPO test accuracy when constrained to using only p=10\% of the available training pool. Specifically, given a pool of unlabeled prompts \mathcal{X} , we select subsets S\subset\mathcal{X} with |S|=\lfloor p|\mathcal{X}|\rfloor according to different difficulty-based policies, train models using GRPO on each subset, and compare their performance on held-out test sets. This setup mirrors practical scenarios where annotation budgets limit the number of examples that can be used for fine-tuning. 2.2 Difficulty estimation via multi-sample probing To robustly estimate example difficulty, we employ multi-sample evaluation using the base model before any fine-tuning. For each prompt x in the training pool, we sample K independent completions from the base model \pi_{\text{base}} using temperature \tau=1.0 and chain-of-thought prompting where applicable [ 26 ] . For each prompt x , we compute: \displaystyle\hat{p}(x) \displaystyle=\frac{1}{K}\sum_{i=1}^{K}\mathbf{1}[\text{completion}_{i}\text{ is correct}] (1) where \hat{p}(x) represents the empirical success rate. Task-specific correctness criteria: • GSM8K: Exact match of the final numerical answer after extracting from the generated solution [ 4 ] • Tracking Shuffled Objects: Correct identification of all object positions after the described swaps [ 22 ] We use K=10 samples for Tracking Shuffle Objects, and K=5 for GSM8K due to its larger size and associated computational costs. This multi-sample approach provides stable difficulty estimates while remaining computationally tractable. 2.3 Subset selection policies Given difficulty estimates for all prompts, we implement four selection policies, each choosing exactly 10\% of the training pool: 1. Hard (est): Select prompts with the lowest success rates: S_{\text{hard}}=\arg\min_{S:|S|=\lfloor p|\mathcal{X}|\rfloor}\sum_{x\in S}\hat{p}(x) 2. Easy (-est): Select prompts with the highest success rates: S_{\text{easy}}=\arg\max_{S:|S|=\lfloor p|\mathcal{X}|\rfloor}\sum_{x\in S}\hat{p}(x) 3. Middle : Select prompts nearest to the median difficulty, specifically those in the interquartile range around the median \hat{p}(x) value 4. Random : Uniform random sample without replacement from the full pool Figure 1 illustrates the complete pipeline from