Verifier-Induced Support Reshaping in On-Policy Optimization
Shaohang Wei 1, Zikun Su 2, Feifan Song 1, Wen Luo 1, Wei Li 1, Guangyue Peng 1 and Houfeng Wang 1 · 2026
Read Verifier-Induced Support Reshaping in On-Policy Optimization by Shaohang Wei 1, Zikun Su 2, Feifan Song 1, Wen Luo 1, Wei Li 1, Guangyue Peng 1 and Houfeng Wang 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00220.txt.
Introduction
Wei and co-authors name a failure mode in on-policy optimization with verifiable rewards. Training can lift the current objective while harming later ones. Successful behaviours for future goals may grow too rare to sample. They call this verifier-induced support reshaping. Effective rewardable support is what a fixed rollout budget can reach.
Across two model families, they track the effect with repeated sampling and two-way training. Math training raised mean instruction success but cut prompts with any success at best-of-32. Instruction training shifted math replies from stepwise openings to direct answers. Those changes concentrate in the first few response tokens. Tested constraints and distillation preserved cross-task support only partly. Endpoint gains do not guarantee future trainability, they warn.
Catalog record
| CREATOR | Shaohang Wei 1, Zikun Su 2, Feifan Song 1, Wen Luo 1, Wei Li 1, Guangyue Peng 1 and Houfeng Wang 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2608.00220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Verifier-Induced Support Reshaping in On-Policy Optimization Shaohang Wei 1, Zikun Su 2, Feifan Song 1, Wen Luo 1, Wei Li 1, Guangyue Peng 1 and Houfeng Wang 1 Abstract We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget. Across two model families, we study this effect through repeated verifier-scored sampling and bidirectional training on mathematical reasoning and constrained instruction following, including sequential training with the opposite verifier. Math-RLVR raises average instruction-following success but reduces the number of prompts with any successful response under repeated sampling. On IFEval with Qwen3-8B-Base, pass@ 1 rises by 6.5 percentage points while best@ 32 falls by 9.8 percentage points, and the same divergence appears across both models and IF benchmarks. Conversely, IF-RLVR shifts math responses from step-by-step openings toward direct answers, lowers best@ k \text{best@}\metricmathit{k} across sampling budgets, and reduces reward variation for later Math-RLVR. Token-distribution analyses and controlled opening interventions show that these changes concentrate in the first few response tokens. RLVR mainly reranks openings already available in the base policy, and the selected opening causally affects math searchability. The tested reference-policy constraints, routing priors, and on-policy distillation preserve cross-task support only partially; MathIF and ReasonIF show that marginal gains translate only partly into responses that are both correct and constraint-following. Therefore, endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. \institution Peking University \reporttype PKU ICL Technical Report \reportnumber PKU-AI-TR-2026-001 \reportdate 2026-07 \reportversion v1.0 \pkuailogofile assets/pku-horizontal-logo.pdf \githubpage https://github.com/sylvain-wei/verifier-induced-support-reshaping \correspondence Houfeng Wang ([email protected]) and Shaohang Wei ([email protected]). \WF@box 1 Introduction Successive post-training stages turn foundation model training into a continual-learning problem. Alignment and reinforcement learning repeatedly adapt a shared policy to objectives such as mathematical reasoning, code generation, and instruction following ( Ouyang et al., 2022 ; Shao et al., 2024 ; Guo et al., 2025a ; Yang et al., 2024 ; Pyatkin et al., 2025 ; Li et al., 2026 ) . Each stage should preserve both existing capabilities and the ability to learn later objectives ( Kirkpatrick et al., 2017 ; Parisi et al., 2019 ) . Recent studies find that on-policy reinforcement learning often retains non-target performance better than supervised fine-tuning at comparable target-task performance ( Shenfeld et al., 2026 ; Chen et al., 2026 ; Lai et al., 2026 ) . However, this evidence is mainly retrospective, asking what performance remains after adaptation. Figure 1 : Verifiers reshape what on-policy training can learn next. Left: Catastrophic forgetting looks back at retained performance; support reshaping looks ahead at future trainability. Middle: Training on task A shifts task B’s initiation support (top) and prompt-end hidden states (bottom). Right: We test both orders with Math-RLVR and IF-RLVR. Continual RLVR therefore has a forward-looking requirement: successful behaviors for later objectives must remain reachable under the current policy. RL with verifiable rewards (RLVR) updates the policy from trajectories sampled from the current policy and scored by a verifier ( Schulman et al., 2017 ; Shao et al., 2024 ) . A successful behavior may remain possible yet become too rare to appear within a finite rollout budget, leaving a later objective with little positive training signal. We call this change support reshaping and define effective rewardable support as reward-positive trajectories that remain likely enough to be sampled under a specified budget. This view differs from catastrophic forgetting: forgetting measures whether performance on a learned task survives later training, whereas support reshaping measures whether successful trajectories for a future task remain discoverable before that task is optimized. As illustrated in Figure 1 , our question is not only what a verifier improves now, but also what it leaves learnable next. We study this requirement through a controlled bidirectional comparison of mathematical reasoning and constrained instruction following. Both are central post-training objectives with programmatically checkable rewards, but they use different success criteria. The math verifier checks final-answer correctness while leaving the solution route open, whereas the IF verifier evaluates compliance with explicit response constraints ( Shao et al., 2024 ; Pyatkin et al., 2025 ) . Prior work has also observed a tension between reasoning and instruction following at evaluation time, making this pair a useful setting for studying cross-task effects ( Fu et al., 2026 ; Li et al., 2025 ) . Starting from the same base policies, we train one branch with Math-RLVR and measure IF support, and train the other with IF-RLVR and measure math support. We then continue each branch with the opposite verifier to test whether the first-stage support shift constrains later learning. This bidirectional design reveals whether the cross-task effect changes with training direction. Our results show that optimizing one verifier changes which successful behaviors the other verifier can still sample and reinforce. Math-RLVR raises IF pass@ 1 but lowers best@ 32 , as more prompts move toward consistent success or complete failure. IF-RLVR lowers math \text{best@}\metricmathit{k} and constrains subsequent Math-RLVR, while math responses shift from step-by-step openings toward direct answers. Distributional analyses and controlled interventions further show that the main change occurs in the first few response tokens, where the opening choice causally affects math searchability. In the tested settings, sequential training, reference-policy constraints, routing priors, and on-policy distillation (OPD) preserve only part of the affected support or trade preservation against target-task gains. Results on MathIF ( Fu et al., 2026 ) and ReasonIF ( Kwon et al., 2026 ) further show that gains in math or instruction following translate only partly into responses that satisfy both. We make the following contributions: • Support reshaping in RLVR. We frame continual RLVR in terms of future trainability and measure effective rewardable support through repeated verifier-scored sampling. Across two model families, our bidirectional Math/IF experiments show that Math-RLVR polarizes IF support, while IF-RLVR reduces math searchability and constrains subsequent Math-RLVR. • A causal role for response openings. We localize the cross-task shift to the first few response tokens and find that RLVR mainly reranks opening options already available in the base policy. Controlled route interventions show that the opening choice causally affects math searchability, favoring a change in route selection over broad erasure of reasoning ability. • Limits of support preservation. We evaluate sequential training, reference-policy constraints, routing priors, and OPD, and find that they only partially preserve future support or trade it against target-task gains. MathIF and ReasonIF further show that improvements in math or instruction following do not translate proportionally into responses that satisfy both. \WF@box 2 Related Work Verifiable Rewards and Reasoning Post-training. Reinforcement learning with verifiable rewards (RLVR) has become an important post-training approach for improving large language models on mathematics, coding, and general reasoning tasks ( Song et al., 2025 ) . Recent reasoning systems show that executable or otherwise checkable rewards can substantially improve performance on the task being optimized ( Shao et al., 2024 ; Guo et al., 2025a ; Kimi Team et al., 2025 ; Yang et al., 2024 ) . Earlier work on learned verifiers, self-consistency, mathematical reasoning benchmarks, and process supervision helped establish how reasoning outputs can be