Self-Trained Verification for Training- and Test-Time Self-Improvement
2026
Read Self-Trained Verification for Training- and Test-Time Self-Improvement (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30290.txt.
Introduction
Chen Henry Wu and Aditi Raghunathan propose self-trained verification (STV) for self-improvement at training and test time, gated by the verifier bottleneck. Loops stall when scores inflate while accuracy stagnates. A model cannot catch its own errors alone, but can when shown the reference solution; STV trains the verifier to imitate that version. At test time, STV improves loops on hard problems where alternatives do not, doubling math accuracy and lifting scientific reasoning fourteenfold. At training time, verifier-in-the-loop training adds a third to pass@1.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39477 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30290
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Trained Verification for Training- and Test-Time Self-Improvement Abstract Self-improvement at scale has been a longstanding goal for reasoning models, and there are two natural places to do it: at test time, through verification-refinement (V-R) loops; and at training time, through self-training methods. Both are gated by the same bottleneck: the verifier. V-R loops stall when verifier scores inflate while accuracy stagnates, and when feedback is too generic to act on; self-training fails similarly when bad self-generated data are added to training. Better verification would unlock both, but the capability we want to train, i.e., catching self-generated errors, lacks training signal. To address this challenge, we propose self-trained verification (STV). Our key observation is that, while a model cannot catch these errors alone, it can when shown the reference solution. We turn this asymmetry into a supervision target and train the verifier to imitate a more informed version of itself. At test time, STV substantially improves V-R loops on hard problems, while alternatives (e.g., SFT, RL on verifier scores, and even meta-verifiers) do not. STV roughly doubles accuracy on hard math and lifts it 14 × 14\times on scientific reasoning tasks ( 1.5 % → 21 % 1.5\%\to 21\% ). At training time, we additionally train the generator using RL with STV verifier’s feedback inside the V-R loop – a procedure we call verifier-in-the-loop training (ViL). Starting from an RL-converged generator, ViL yields a further 33 % 33\% gain in pass@1. More notably, the generator’s standalone pass@1, with no verifier at test time, climbs 30 % 30\% relative past where standard RL had converged. Hence, the next frontier in reasoning on hard problems may lie in how we train for and with verification. 1 Introduction Figure 1: (a) We propose self-trained verification (STV), a scalable recipe for training verifiers that judge solutions and provide feedback for generators to refine. STV unlocks both self-improvement for test-time iterative refinement and training-time generator improvement. (b) STV scales much better with test-time compute than untrained or verdict-only-RL verifiers. (c) After standard RLVR converges, training the generator with verifier feedback in the loop (ViL) yields further pass@1 gain, along with a standalone gain (shaded) visible without any verification at test time. The long-term promise of reasoning models is self-improvement, where models can use their own attempts, mistakes, and feedback to solve harder problems. There are two natural places to do it. At test time, verifier-guided refinement ( 13 ; 29 ) turns additional inference compute into better solutions, where a generator and verifier alternate, with the generator producing a solution, the verifier providing feedback to it, and the generator refining. This drives recent breakthroughs on the hardest reasoning tasks like IMO and frontier math ( 8 ) . At training time, self-training ( 40 ; 32 ) recycles the generator’s own, verified attempts into supervision. While recent frontier systems lean increasingly on verification, their gains are capped by verifier quality: shallow critiques ( 12 ; 16 ; 35 ; 34 ) and miscalibrated scores ( 5 ; 23 ) cause additional compute to amplify plausible but wrong reasoning. In this work, we develop a scalable recipe for training verification and generation that unlocks self-improvement at both ends: test-time iterative refinement (by 14\times on hard reasoning tasks) and training-time generator improvement (by 30\% beyond the RLVR plateau). Final-answer labels can teach a verifier to judge whether a solution is wrong, but not to find the flaw in a plausible but incorrect one – a missed boundary case, an invalid reduction, or a lemma used outside its hypotheses. Existing systems sidestep this supervision gap with meta-verifiers trained on human feedback ( 29 ) or with external feedback such as a stronger model 34 . We instead exploit a simple asymmetry: diagnosis is easier given a reference solution . A model that struggles to find the flaw on its own can often locate it when shown a reference, because the task shifts from solving the problem to comparing a candidate against a known-good solution. Our pipeline, self-trained verification (STV), turns this privileged setting into supervision. A reference-conditioned verifier acts as a teacher, and on-policy distillation trains an unconditioned verifier to match its feedback distribution. At test time, the resulting verifier runs without references. STV makes verifier-guided refinement effective on problems out of reach for the base generator. On hard DAPO math problems ( 39 ) – including a Hardest split where Qwen3-8B has zero pass@1– an STV verifier roughly doubles final-round pass@1 over an untrained verifier, with gains compounding over 20+ refinement rounds. Standard alternatives (SFT on reference-conditioned feedback, RL on verdict accuracy) saturate quickly and yield much smaller gains (Figure 1 , middle). The gains extend beyond math: on the hardest SciKnowEval problems, STV lifts pass@1 from 1.5% without verification to 21.0%. With enough verification compute, Qwen3-8B guided by STV outperforms the much larger Qwen3-32B generator, suggesting that trained verification can substitute for generator scale on hard reasoning problems. Beyond test-time refinement, STV also enables a new way to training-time self-improvement: training the generator inside the V-R loop with the trained verifier providing feedback, a procedure we call verifier-in-the-loop training (ViL). Starting from a generator already converged under standard reinforcement learning from verifiable reward (RLVR), we continue training it with feedback from a frozen STV verifier. Unlike standard self-training, the training signal here remains verifiable since the verifier’s feedback only provides additional context that helps the generator to maximize that reward. ViL unlocks gains beyond the RLVR ceiling: final-round pass@1 improves by 33\% relative with the verifier present, and standalone pass@1 improves by 30\% relative even with no verifier at inference. However, the same compute spent extending standard RLVR yields no further gain (Figure 1 , right, shaded). In summary, STV improves both modes of verifier-driven self-improvement: at test time, it makes refinement scale; at training time, it breaks through the standard RLVR plateau, with gains that persist even with no verifier at inference. Our results point to a shift in where reasoning gains come from: from learning to verify – and using stronger verifiers to train stronger generators. This suggests an iterative route to self-improvement, where better verifiers enable more reliable test-time refinement and richer generator training, while better generators produce harder attempts for future verifier training. Developing scalable ways to train verifiers without human-graded feedback may therefore be a key frontier for reasoning models that improve at both test time and training time. 2 Problem Formulation We consider a reasoning problem x with ground-truth answer y^{\star}(x) . A verification-refinement (V-R) pipeline couples a generator G with a verifier V . These may be two separate models or the same model invoked with different prompts; we instantiate both cases in our experiments. At round 0 , the generator produces an initial solution y_{0}\sim G(\cdot\mid x) . At each subsequent round r\geq 1 , the verifier examines the latest solution and returns a verdict together with natural-language feedback, (v_{r},f_{r})\sim V(\cdot\mid x,y_{r-1}),\qquad v_{r}\in\{\text{accept},\text{reject}\}. If v_{r}=\text{reject} , the generator refines the solution based on the feedback: y_{r}\sim G(\cdot\mid x,y_{r-1},f_{r}) . The loop ends at the first acceptance or after R rounds. 3 Self-Trained Verification We propose self-trained verification (STV), which trains this capability without human annotation: while a model cannot