Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs
Suraj Yadav, Siddharth Yadav, Parth Goyal · 2026
Read Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs by Suraj Yadav, Siddharth Yadav, Parth Goyal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06298.txt.
Introduction
Yadav, Yadav, and Goyal test preference optimization in small models. They apply GRPO with LoRA to models up to 3 billion parameters. As difficulty rises, accuracy hits a capacity boundary: GRPO reshapes preferences without reliably improving the hardest tier. Training only on easier problems matches full accuracy using about 45 percent of steps, showing diminishing returns from harder samples. GSM8K-trained beat MATH-trained on MATH's numeric subset by about 5 percent at 1.5 billion and 3 percent at 3 billion. Best gains depend on base-model reasoning skill and dataset difficulty.
Catalog record
| CREATOR | Suraj Yadav, Siddharth Yadav, Parth Goyal |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72437 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.06298
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs Suraj Yadav, Siddharth Yadav, Parth Goyal Abstract Recent alignment work on Large Language Models (LLMs) suggests preference optimization can improve reasoning by shifting probability mass toward better solutions. We test this claim in a resource-constrained setting by applying GRPO with LoRA to SLMs (0.5B–3B) for math reasoning on GSM8K and MATH datasets with difficulty-stratified analyses. As problem difficulty increases, accuracy plateaus, revealing a capacity boundary: GRPO primarily reshapes output preferences without reliably improving hardest-tier solving. Consistent with this, training GRPO only on lower-difficulty problems matches full-dataset accuracy across difficulty tiers while using only ∼ 45 % \sim 45\% training steps, indicating diminishing returns from harder samples in this regime. We also find a cross-dataset generalization effect: GSM8K-trained GRPO achieves higher accuracy on the numeric subset of MATH than MATH-trained GRPO, exceeding it by ∼ 5 % \sim 5\% at 1.5B and by ∼ 3 % \sim 3\% at 3B. We show that the best achievable gains depend strongly on the base model’s prior reasoning competence and the dataset’s difficulty profile. \DeclareCaptionType sample[Sample][List of Completions] 1 Introduction Figure 1: Comparison of accuracy on the MATH dataset for Qwen2.5 0.5B, 1.5B, and 3B models Qwen et al. 2025 . Models trained on the lower-difficulty subsets (L1–L3) achieve comparable or superior performance to those trained on the full dataset (L1–L5). Group Relative Policy Optimization (GRPO) Shao et al. 2024 , has proven effective for enhancing mathematical reasoning in large language models (LLMs). However, while SLMs often inherit reasoning via distillation from larger teachers Guo et al. 2025 , the efficacy of applying GRPO directly in resource-constrained regimes remains under-explored. Current literature generally suggests that exposure to higher-difficulty training data is associated with improved reasoning performance Tong et al. 2024 , particularly in LLMs. In compute resource constraint settings, it is critical to determine if Small Language Models (SLMs) possess the intrinsic capacity Wei et al. 2022 to benefit from complex alignment samples, or if they face a capacity boundary. Capacity boundary is a threshold below which algorithmic improvements cannot overcome the fundamental lack of model expressivity required for complex reasoning. In this work, we test this hypothesis by conducting a difficulty-stratified Ding et al. 2025 Gao et al. 2025 analysis of GRPO on SLMs. We fine-tune models ranging from 0.5B to 3B parameters using LoRA Hu et al. 2021 on the GSM8K Wei et al. 2022 and MATH Tong et al. 2024 datasets. Unlike standard evaluations, we perform a difficulty-stratified analysis, isolating the impact of training data complexity on downstream performance across different tiers of problem hardness. 1. Identification of the Capacity Boundary: We provide empirical evidence that GRPO-based alignment hits a hard saturation point in SLMs, preventing further learning on complex reasoning tasks. 2. Efficiency via Difficulty Stratification: We demonstrate that filtering out ”hard” training samples does not degrade performance. Training on only the lower-difficulty \sim 45% of the dataset achieves competitive results with full-dataset training, offering a protocol for significantly more efficient alignment. 3. Cross-Dataset Generalization Analysis: We analyze the transfer capabilities between models trained on GSM8K and MATH, identifying cases where ”easier” training distributions (GSM8K) provide a stronger signal for numeric reasoning than domain-specific ”hard” distributions (MATH). 2 Related Work 2.1 Post-Training Alignment and GRPO Post-training alignment methods like RLHF Ouyang et al. 2022 , DPO Rafailov et al. 2024 , and specifically Group Relative Policy Optimization (GRPO) Shao et al. 2024 effectively enhance reasoning in Large Language Models but remain under-explored for Small Language Models (SLMs). Unlike traditional RLHF, which relies on a separate, memory-intensive value function to compute policy baselines, GRPO samples a group of G outputs \{o_{1},\dots,o_{G}\} for a single prompt q . It then computes the baseline directly from this group’s empirical mean. The optimization objective is formulated as: \begin{split}\mathcal{J}_{\text{GRPO}}&(\theta)=\mathbb{E}_{q\sim\mathcal{D},o\sim\pi_{\theta_{\text{old}}}}\\ &\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\min(r_{i,t}(\theta)\hat{A}_{i,t},\text{clip}(r_{i,t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{i,t})-\beta D_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})\right]\end{split} (1) where r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}. where the advantage \hat{A}_{i,t} is the normalized reward of output o_{i} relative to its group. This mechanism significantly reduces memory overhead and naturally incentivizes the model to outperform its own average generation, a dynamic well-suited for iteratively improving mathematical Chain-of-Thought reasoning. 2.2 Data Complexity and the Capacity Boundary Furthermore, while existing curricula prioritize high-complexity data Ding et al. 2025 , we demonstrate that for capacity-constrained SLMs, such strategies often induce negative transfer. Our work investigates this “capacity boundary,” establishing that simpler signals yield superior generalization in sub-3B regimes. (See Appendix A for extended review). 3 Methodology Training Framework We adopt a two-stage training protocol. Models Qwen 2.5 0.5B, 1.5B, 3B Qwen et al. 2025 first undergo a brief phase of Supervised Fine-Tuning (SFT) to establish instruction-following behavior and baseline reasoning capabilities. Subsequently, we align these SFT-initialized models using Group Relative Policy Optimization (GRPO) via the TRL library von Werra et al. 2020 . To ensure parameter efficiency, we apply Low-Rank Adaptation (LoRA) to all linear projection layers utilizing the PEFT library Mangrulkar et al. 2022 . Detailed hyperparameters for both SFT and GRPO stages are provided in Appendix B . Difficulty-Aware Reward Modeling We design a composite reward function that enforces both strict XML-based formatting and reasoning correctness. A key component of our methodology for the MATH dataset is difficulty scaling: we dynamically weight correctness rewards based on the problem’s difficulty level. This ensures that successful reasoning on complex tasks provides a stronger optimization signal. Dataset & Stratification We utilize the GSM8K dataset and a targeted selection of the Hendrycks MATH benchmark, specifically focusing on four core domains: algebra , counting_and_probability , geometry , and number_theory . To verify our hypothesis regarding capacity boundaries, we map these into a unified two-tier complexity hierarchy: • Low Difficulty: GSM8K (grade-school arithmetic), MATH Levels 1–3. • High Difficulty: MATH Levels 4–5. Evaluation Protocol We evaluate all models using greedy decoding ( T=0 ) to assess deterministic reasoning capabilities. We benchmark performance improvements directly against the SFT-tuned baselines (the model state prior to GRPO alignment). Our critical analysis focuses on upward transfer : measuring whether models trained exclusively on ”Low” data partitions can generalize to ”High” difficulty test samples without direct exposure to them. Additionally, we extend this evaluation to both GSM8K-trained and MATH-trained models, testing them on the numerical-answer subset of the full MATH test set (see Appendix B.2 ). 4 Experimental Results 4.1 The Capacity Boundary: Saturation on High-Difficulty Tasks We investigate whether exposing small models to high-difficulty reasoning paths improves general performance. Figure 1 compares models tuned on the full MATH dataset (Levels 1–5) versus the easier subset (Levels 1–3). We observe that including high-difficulty problems yields no significant gain across any model size. In fact, for the 0.5B and 1.5B models, the inclusion of Level 4–5 data proves detrimental: the 0.5B model achieves 21.5% accuracy on the L1–L3 subset but drops to 18.7% when trained on the full dataset. Similarly, the 1.5B model sees a regression from 38.4% to 36.6% with the addition of harder tasks. Even for the 3B model, performance saturates, showing negligible difference between conditions (46.9% vs. 47.0%). Crucially, to verify that this saturation is not merely an optimization artifact of parameter-efficient fine-tuning, we conducted a full fine-tuning (FFT) ablation on the 0.5B model. The FFT results mirrored the LoRA findings, showing a similar