Takeaway: Co-Training Generation and Pairwise Self-Verification
Unified RL training that jointly optimizes generation and pairwise verification ( · 2026
Read Takeaway: Co-Training Generation and Pairwise Self-Verification by Unified RL training that jointly optimizes generation and pairwise verification ( (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04304.txt.
Introduction
Singh and colleagues attack verification, the bottleneck when reasoning is scaled by sampling many solutions. Pointwise scoring rates each answer alone, but models prove stronger when comparing candidates in pairs. Their V1 framework unifies generation and self verification through pairwise ranking. V1 Infer runs a tournament spending effort where rankings are most uncertain, while V1 PairRL trains one model to generate and verify in pairs. On coding and mathematics benchmarks, inference lifts Pass at 1 by up to 10 percent over pointwise verification, with further trained gains.
Catalog record
| CREATOR | Unified RL training that jointly optimizes generation and pairwise verification ( |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89801 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04304
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Takeaway: Co-Training Generation and Pairwise Self-Verification Unified RL training that jointly optimizes generation and pairwise verification ( Abstract Test-time scaling for complex reasoning tasks shows that leveraging inference-time compute, by methods such as independently sampling and aggregating multiple solutions, results in significantly better task outcomes. However, a critical bottleneck is verification : sampling is only effective if correct solutions can be reliably identified among candidates. While existing approaches typically evaluate candidates independently via scalar scoring, we demonstrate that models are substantially stronger at pairwise self-verification . Leveraging this insight, we introduce V 1 \textbf{V}_{1} , a framework that unifies generation and verification through efficient pairwise ranking. V 1 \textbf{V}_{1} comprises two components: V 1 \textbf{V}_{1} -Infer , an uncertainty-guided algorithm using a tournament-based ranking that dynamically allocates self-verification compute to candidate pairs whose relative correctness is most uncertain; and V 1 \textbf{V}_{1} -PairRL , an RL framework that jointly trains a single model as both generator and pairwise self-verifier, ensuring the verifier adapts to the generator’s evolving distribution. On code generation (LiveCodeBench, CodeContests, SWE-Bench) and math reasoning (AIME, HMMT) benchmarks, V 1 \textbf{V}_{1} -Infer improves Pass@1 by up to 10 % 10\% over pointwise verification and outperforms recent test-time scaling methods while being significantly more efficient. Furthermore, V 1 \textbf{V}_{1} -PairRL achieves 7 7 – 9 % 9\% test-time scaling gains over standard RL and pointwise joint training, and improves base Pass@1 by up to 8.7% over standard RL in a code-generation setting. \textbf{V}_{1} : Unifying Generation and Self-Verification for Parallel Reasoners Harman Singh ∗ 1 Xiuyu Li ∗ 1 Kusha Sareen 2 Monishwaran Maheswaran 1 Sijun Tan 1 Xiaoxia Wu 3 Junxiong Wang 3 Alpay Ariyak 3 Qingyang Wu 3 Samir Khaki 1 Rishabh Tiwari 1 Long Lian 1 Yucheng Lu 3 Boyi Li 1,4 Alane Suhr 1 Ben Athiwaratkun 3 Kurt Keutzer 1 1 UC Berkeley 2 Mila 3 Together AI 4 NVIDIA Project Page | Code Correspondence to: {harmans, xiuyu, keutzer}@berkeley.edu ∗ Equal contribution. 1 Introduction Large language models (LLMs) have demonstrated remarkable problem-solving abilities, largely driven by the paradigm of “System 2” thinking [ 28 , 29 , 6 ] of generating extended chains of thought to reflect, refine, and verify answers at inference time. Parallel reasoning , which complements this sequential “deep thinking” by sampling multiple independent chains of thought to explore diverse solution paths, has emerged as a powerful technique for test-time scaling [ 45 , 4 , 30 , 17 , 39 ] . In this setup, parallel sampling of multiple chains-of-thought is followed by an aggregation step to select the final answer. The simplest form of aggregation is to select the most common solution among the set of candidates (majority voting). While this suffices for domains like math where answers are objective and easily verifiable [ 45 ] , this can not be used for more general domains which do not admit objective ground truth answers. Instead, the ability to accurately self-verify solutions can support an aggregation method that selects the correct solution from the candidate set, as long as it exists in the set, even if it isn’t the most common solution. Thus, taking full advantage of parallel reasoning fundamentally hinges on accurate self-verification : sampling N solutions is useful if the model can reliably identify the correct one. Our experiments identify a critical bottleneck in existing approaches that use models’ intrinsic verification capabilities to take advantage of inference-time compute: without a globally comparable scale of solution quality, existing models are not calibrated to evaluate candidate solutions independent of one another. In addition, existing work suggests that in such settings, models used as verifiers are biased towards positively evaluating their own samples, even if those samples are incorrect [ 21 ] . We find that, instead, self-verifying between candidate solutions via pairwise comparison leads to more robust and accurate outcomes. We explore two core questions. First, how can we enable LLMs to more accurately self-verify candidate solutions obtained via parallel reasoning, by leveraging pairwise candidate self-verification? Second, given that self-verification is typically applied after a model has already been trained, can we instead train models to be better at self-verification for parallel reasoning? While pairwise ranking has been extensively studied in reward modeling for LLM alignment [ 3 , 59 ] , this approach remains underexplored for self-verification in a parallel reasoning setting. Additionally, while reinforcement learning is commonly used to improve the solution-generation capabilities of LLMs on verifiable domains such as code and math [ 35 , 6 ] , no existing methods effectively utilizes the parallel reasoning chains of LLMs at training time to jointly optimize both the generation and self-verification capabilities, resulting in distribution shifts at inference time. We demonstrate that inducing pairwise self-verification capabilities during RL training is an effective technique to improve test-time scaling performance for parallel reasoners. We leverage our observations to develop \textbf{V}_{1} , a unified framework that includes a strong inference time scaling algorithm for parallel reasoning, as well as a reinforcement learning framework that induces self-verification capabilities during RL training of LLMs with verifiable rewards. Our contributions include the following: 1. We show that in parallelized reasoning, independent self-verification of candidate solutions suffers from calibration collapse due to lack of comparative reference. On the other hand, self-aggregation methods like recursive self-aggregation [ 44 , RSA; ] may lead to diversity collapse where Pass@N monotonically decreases with aggregation steps. This motivates pairwise verification as a principled alternative for self-verification and an orthogonal methodology for test-time scaling without loss of diversity. 2. We develop \textbf{V}_{1} -Infer , an uncertainty-guided pairwise verification algorithm. Rather than scoring solutions in isolation, it pairs candidates by employing a Swiss-system tournament refinement strategy that dynamically allocates verification compute to the most uncertain pairs. This approach provides a significant boost in selection accuracy, effectively improving the performance of the model closer to Pass@N of the original sampled responses. Notably, we find that \textbf{V}_{1} -Infer outperforms or matches RSA on tasks where aggregation leads to diversity collapse, while requiring significantly fewer verification calls. 3. We develop \textbf{V}_{1} -PairRL , an RL framework that co-trains a single model as both generator and pairwise self-verifier. Unlike prior co-training approaches that rely on pointwise rewards [ 33 , 18 ] or offline data [ 44 ] , \textbf{V}_{1} -PairRL uses an online, co-evolving objective where generation and pairwise verification improve together. This ensures that as the generator improves, the verifier trains on in-distribution data from the model’s current capabilities, thereby leading to stronger self-verification capabilities at inference time. We evaluate our framework on code generation (LiveCodeBench, CodeContests) and math reasoning (AIME, HMMT) benchmarks. \textbf{V}_{1} -Infer improves Pass@1 by up to 10% over pointwise verification and matches or exceeds RSA. \textbf{V}_{1} -PairRL achieves 7–9% test-time scaling gains over pointwise co-training standard RL, and improves base Pass@1 by up to 8.7% over the standard RL. Our results demonstrate that unified training for solution generation and self-verification capability, coupled with our pairwise self-verification technique, enables effective parallel reasoning. 2 Related Work Parallel reasoning and test-time scaling. Test-time scaling typically follows two paradigms: sequential refinement [ 23 ] or parallel generation of multiple reasoning paths [ 4 , 34 , 30 ] . While parallel scaling allows for