Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning
Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi · 2026
Read Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning by Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15830.txt.
Introduction
Fang and four colleagues propose PieceHint, a training framework that helps language models learn reasoning by giving hints exactly where they matter. Rather than handing out uniform partial solutions, it scores reasoning steps, places hints at bottlenecks matched to problem difficulty, then gradually withdraws support toward independent work. On six mathematical reasoning benchmarks, their 1.5B model matched much larger 32B baselines on average while preserving diverse correct answers.
Catalog record
| CREATOR | Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52964 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15830
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi Abstract Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma: training on easy problems can cause overfitting and pass@k degradation, while training on hard problems often results in sparse rewards. Recent question augmentation methods address this by prepending partial solutions as hints. However, uniform hint provision may introduce redundant information while missing critical reasoning bottlenecks, and excessive hints can reduce reasoning diversity, causing pass@k degradation. We propose PieceHint , a hint injection framework that strategically identifies and provides critical reasoning steps during training. By scoring the importance of different reasoning steps, selectively allocating hints based on problem difficulty, and progressively withdrawing scaffolding, PieceHint enables models to transition from guided learning to independent reasoning. Experiments on six mathematical reasoning benchmarks show that our 1.5B model achieves comparable average performance to 32B baselines while preserving pass@k diversity across all k k values. † † footnotetext: Authors are listed in no particular order. \dagger Corresponding author. 1 Introduction Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing LLM reasoning capabilities 9 ; 28 . However, RL-based training faces a fundamental dilemma: training on easy problems causes overfitting and pass@k degradation, while training on hard problems yields sparse reward signals 47 ; 33 . Figure 1: RL training dilemma and PieceHint solution. Top : Missing key insights at reasoning bottlenecks lead to sparse/zero rewards. Bottom : PieceHint identifies critical pieces from ground-truth solutions, injects them as hints to bridge gaps, enabling informative rewards and pass@k diversity. Various approaches address this challenge through reward shaping 54 ; 32 ; 37 ; 38 , process-level supervision 36 ; 27 , and curriculum learning 24 ; 29 ; 41 ; 20 . Question augmentation offers a complementary direction: StepHint 48 prepends partial ground-truth solutions as hints, enabling models to discover successful trajectories on hard problems. However, uniform hint provision may introduce redundant information while missing critical bottlenecks, and excessive hints can reduce reasoning diversity, causing pass@k degradation. Figure 1 illustrates the core challenge and our solution. In reinforcement learning, models receive rewards only upon generating correct final answers—partial progress yields no credit. When facing hard problems, models often fail at specific reasoning bottlenecks—missing a pivotal lemma, overlooking a non-obvious insight, or lacking a critical technique. These bottlenecks act as missing puzzle pieces: without them, the reasoning chain cannot be completed. If all attempts fail at these bottlenecks, reward variance becomes zero, eliminating gradient signals for learning (Figure 1 , top). Uniform hint provision may address routine derivations the model can already handle, while leaving critical reasoning gaps unbridged. This reveals a key insight: reasoning breakthroughs occur at specific bottlenecks rather than uniformly across steps, and identifying and providing these critical puzzle pieces yields maximum learning benefit. To address this challenge, we propose PieceHint , which treats critical reasoning steps as puzzle pieces that unlock solution pathways (Figure 1 , bottom). Our approach strategically identifies high-value pieces through value-driven scoring, allocates them based on problem difficulty and model capability, and progressively withdraws them during training, enabling models to transition from scaffolded learning to independent reasoning. Our contributions are: • We propose PieceHint, a hint injection framework that identifies critical reasoning pieces and strategically provides them during training. • We introduce progressive piece withdrawal curriculum, enabling transition from scaffolded to hint-free reasoning. • Experiments on six mathematical benchmarks show our 1.5B model achieves comparable average performance to 32B baselines while demonstrating superior parameter efficiency and preserving exploration diversity. 2 Related Works Reinforcement Learning for Reasoning. Reinforcement learning has emerged as a powerful paradigm for enhancing language model reasoning capabilities, particularly in mathematical problem-solving 9 ; 28 . Foundational algorithms such as PPO 31 and GRPO 9 establish the algorithmic basis for policy optimization with outcome supervision. Subsequent work has explored orthogonal improvements across multiple dimensions: reward shaping to provide intermediate signals 54 ; 32 , process-level supervision to credit individual reasoning steps 36 ; 27 , and chain-of-thought verification to validate reasoning trajectories 39 . Sample Efficiency and Exploration. Parallel efforts address sample efficiency and exploration through uncertainty-aware training strategies. TreeRL 13 and VinePPO 18 leverage entropy or confidence metrics to prioritize high-quality rollouts and improve credit assignment. MRT 30 reuses partial trajectories at inference time to amortize computational cost across multiple queries 51 . Curriculum learning strategies have also been proposed to organize training progression, either by solution length 24 , problem difficulty 29 , or reasoning complexity 41 . The Dilemma of Problem Difficulty in RL Training. Recent work 33 ; 47 ; 50 reveals a fundamental tension in RL-based reasoning: training on easy problems can lead to pass@k degradation as models may overfit to familiar solution patterns, while training on hard problems often results in sparse reward signals. To mitigate pass@k degradation, several methods 44 ; 22 employ entropy regularization through adaptive clipping, temperature tuning, or periodic reference model updates. These algorithmic modifications operate at the optimization level, adjusting how models learn from existing rollouts. Question Augmentation for RL. Question augmentation operates at the input level to reshape problem difficulty. StepHint 48 prepends sequential partial solutions (e.g., the first 50% of reasoning steps), bridging the gap between easy and hard problems while improving convergence speed and pass@k preservation. An alternative approach 40 provides entire reasoning chains from stronger models as reference trajectories, though the target model still explores independently most of the time. While these methods demonstrate effectiveness, position-based selection may include routine steps while potentially missing non-obvious insights that appear later in solutions, as validated by our ablation studies (Section 5.3 ). In contrast to position-based approaches, our method identifies critical reasoning bottlenecks through value-driven piece identification and scoring, strategically providing hints based on their impact on learning rather than sequential position. Furthermore, through progressive piece withdrawal, we enable models to transition from scaffolded learning to hint-free reasoning, addressing potential generalization limitations of fixed augmentation methods. Figure 2: Overview of PieceHint. We identify critical reasoning pieces via value scores, allocate them by model capability, and progressively withdraw hints during training for hint-free reasoning. 3 Preliminaries Problem Formulation. We consider mathematical reasoning as a sequence generation task. Given a problem q from problem space \mathcal{Q} , the goal is to generate a reasoning trajectory o that leads to the correct answer. We denote the ground-truth solution as \tau^{*}=[\tau^{*}_{1},\tau^{*}_{2},\ldots,\tau^{*}_{n}] , which is a sequence of reasoning steps, and the final answer as a^{*} . A generated response o is considered correct if its extracted answer matches a^{*} . In question augmentation, we construct augmented prompts by prepending auxiliary information as hints to the original problem. Formally, an augmented prompt is denoted as \tilde{q}=[h;q] , where h represents the hint and [;] denotes concatenation. The hint h can take various forms depending on the augmentation strategy—for instance, partial solution steps, key intermediate results, or critical reasoning components. During RL training, the model learns from augmented prompts \tilde{q} while being evaluated on original prompts q without any hints, ensuring the learned policy generalizes to hint-free scenarios. Group Relative Policy Optimization. We adopt Group Relative Policy