Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification Jiahui Li *, Jianfeng Shan *, Wenpei Chen, Shunyu Wu, Jian Lou, Wenjie Feng, Dan Li, See-Kiong Ng Abstract Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models (LLMs) in a completely label-free manner. Despite existing studies focusing on Pass@1 performance, optimizing Pass@k remains under-explored yet critical in label-free settings, which measures generation coverage for sustained exploration. Optimizing Pass@k in label-free setting is highly non-trivial, as directly applying the Pass@k advantage designs effective for RLVR yields unsatisfactory performance. Through in-depth empirical analysis, we discover the root causes hindering performance: pseudo-label estimations for low-confidence samples have a high probability of being incorrect, while candidate answers for high-confidence samples suffer from severe diversity collapse. To overcome these hurdles, we propose TTRL-CoCoV ( T est- T ime R einforcement L earning with Co nfidence- Co nditioned V erification), a novel confidence-adaptive framework that expands Pass@k coverage and improves Pass@1 performance. Based on our key insight that verification capability generally leads generation capability, TTRL-CoCoV employs a confidence-conditioned mechanism: for high-confidence samples, it bootstraps verifier and applies an exploration-enhancing reward to prevent diversity collapse; for low-confidence samples, it delegates pseudo-label selection to the verifier to filter incorrect pseudo-labels; and for medium-confidence samples, it bypasses verification entirely. Extensive experiments demonstrate that TTRL-CoCoV outperforms the best competing methods across 6 widely-recognized benchmarks, achieves average absolute gains of +9.8% in Pass@1 and +18.7% in Pass@16 over TTRL, and even achieves absolute Pass@1 improvements of up to +5.0% across multiple reasoning benchmarks when compared against fully supervised RL methods. Our code repository: https://github.com/shanjf666/CoCoV . 1 1 footnotetext: Equal contribution. 1 Introduction Test-time reinforcement learning (TTRL) ( Zuo et al. 2025 ; Liu et al. 2025 ; Wang et al. 2025a ; Zhou et al. 2025 ) has emerged as a promising paradigm to enhance the reasoning capabilities of large language models (LLMs) in completely label-free settings. Compared to traditional Reinforcement Learning from Verifiable Rewards (RLVR) ( Jaech et al. 2024 ) ( Guo et al. 2025 ) , which fundamentally relies on human annotations or oracle-provided labels, TTRL optimizes reasoning policies entirely through self-sampling and internal reward construction. Building upon standard architectures such as Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) , recent TTRL methods effectively address the scalability bottlenecks of RLVR and achieve single-pass generation (Pass@1) performance that closely approaches fully supervised baselines. Beyond Pass@1 accuracy, maintaining a broad exploratory space is critical to prevent premature convergence during policy optimization. In the RLVR setting, recent studies ( Chen et al. 2025a ; Walder and Karkhanis 2025 ) have demonstrated that explicitly optimizing for Pass@k generation coverage effectively balances exploration and exploitation. However, despite its proven importance in supervised environments, optimizing Pass@k generation coverage remains largely under-explored within the TTRL setting. To advance the efficacy of Test-Time Reinforcement Learning (TTRL) ( Zuo et al. 2025 ) , current methodologies mainly focus on establishing the reliability of self-generated pseudo-labels. For complex, low-consistency problems, recent sample-level advancements successfully employ consensus gating and confidence reweighting to secure label purity and stabilize the optimization process ( Yan et al. 2026 ; Yu et al. 2025a ) . Alternatively, other approaches leverage internal self-verification mechanisms or an outer verifier to actively filter out spurious trajectories or perform autonomous error correction ( Pan et al. 2026 ; Yan et al. 2026 ; Liao et al. 2026 ) . Simultaneously, researchers have also recognized the importance of preventing premature convergence—often observed as a dramatic collapse in the standard deviation of response lengths on mastered problems ( Chen et al. 2025a ; Walder and Karkhanis 2025 ) . To address this, recent studies have thoughtfully introduced novelty or entropy bonuses to maintain a broad exploratory space in unsupervised TTRL ( Liu et al. 2025 ; Zhou et al. 2025 ) . Collectively, these pioneering efforts have advanced TTRL along two directions: managing label reliability and enhancing exploration. Yet, a fundamental dimension remains surprisingly unexplored: in completely unsupervised TTRL, how to systematically optimize Pass@k generation coverage, which is a critical metric for sustained exploration. This gap is non-trivial, as we will show through three empirical insights below. Directly extending standard Pass@k optimization to the label-free TTRL environment faces three fundamental challenges, which we identify through systematic empirical analysis. First , the challenge of noisy exploration signals: Pass@K optimization inherently relies on exploring low-confidence samples to broaden the exploratory space ( Chen et al. 2025a ) . However, in completely unsupervised TTRL, this introduces severe pseudo-label noise that corrupts the required exploration signals in Pass@K optimization. Second , the challenge of length collapse: While Pass@k successfully sustains response diversity in supervised RLVR, our analysis reveals that applying it in TTRL fails to prevent a severe collapse in response length standard deviation, which acts as a latent leading indicator of premature accuracy stagnation. Third , the challenge of empirical formulation failure: Consequently, directly adopting standard supervised Pass@k advantage formulations fails to expand the exploratory space in label-free settings, causing actual Pass@k performance to inevitably degrade over time. (A comprehensive evaluation and proof of these empirical challenges are provided in Section 3 ). Motivated by these empirical findings, we propose TTRL-CoCoV ( T est- T ime R einforcement L earning with Co nfidence- Co nditioned V erification), a novel framework featuring targeted mechanisms to explicitly resolve each identified challenge. Corresponding to the challenge of noisy exploration signals, we introduce a Confidence-Conditioned Classifying mechanism. By adaptively allocating internal verification resources based on per-sample confidence, it drives co-evolution under a unified optimization objective: bootstrapping the verifier on high-confidence successes to actively filter spurious pseudo-labels on low-confidence hard problems, while safely bypassing medium-confidence boundaries. To overcome the persistent trajectory collapse and the empirical failure of naive Pass@k adaptations, we design an exploration-enhancing reward mechanism ( R_{div} ). This tailored mechanism explicitly incentivizes solution-strategy diversity exclusively on high-confidence problems, successfully adapting the core benefits of Pass@k optimization to the label-free setting without corrupting the pseudo-label pool. We summarize our primary contributions as follows: • Empirical Diagnosis of Label-Free Pass@k Dynamics: We are the first to systematically investigate the under-explored gap of Pass@k generation optimization within completely unsupervised TTRL. Through rigorous empirical analysis, we identify the fundamental bottlenecks, including noisy exploration signals, latent diversity collapse, and formulation mismatches that make this adaptation highly non-trivial in label-free environments. • A Novel Confidence-Adaptive Framework (TTRL-CoCoV): To overcome these challenges, we propose TTRL-CoCoV. It employs a Confidence-Conditioned Classifying mechanism that drives co-evolution between the generator and verifier under a unified optimization objective, effectively filtering noisy exploration signals. Building upon this purified signal, we successfully adapt Pass@k optimization to the unsupervised setting via a tailored exploration-enhancing reward, safely sustaining broad exploratory exploration without corrupting the pseudo-label pool. • Comprehensive Evaluations and SOTA Performance: We conduct exhaustive experiments across six complex reasoning benchmarks (e.g., MATH500, AIME24/25) using multiple backbone models (Qwen3-4B/8B). Empirically, TTRL-CoCoV completely reverses the unsupervised Pass@k degradation, yielding massive exploratory generation coverage (an average absolute gain of +18.7% in Pass@16). Furthermore, via a two-stage annealing strategy, our fully unsupervised method achieves absolute Pass@1 improvements of up to +5.0%, successfully breaking the performance ceiling established by fully supervised GRPO baselines. 2 Backgrounds and Preliminary Problem Setup. Let \mathcal{D} be a dataset of reasoning questions. For a given prompt \boldsymbol{x}\sim\mathcal{D} , a target large language model (LLM) policy \pi_{\theta} generates a response sequence \boldsymbol{y}=(y_{1},\ldots,y_{|\boldsymbol{y}|}) autoregressively. Specifically, \pi_{\theta}(y_{t}\mid\boldsymbol{x},\boldsymbol{y}_{0.5 are admitted into the trusted set \mathcal{A}_{true} . We then select the answer with the highest first-stage consensus from \mathcal{A}_{true} as the pseudo-label. The verifier’s reward is computed based on whether its prediction matches this dynamically mined label. If \mathcal{A}_{true}=\emptyset , we assume that none of the Top- K candidates is likely to be correct and skip the update for this sample. When computing the verifier’s policy loss \mathcal{L}_{second} , we adopt the principle of “lenient to false negatives while strict with false positives”. The primary function of the verifier is not to directly select the correct answer, but to intercept potentially erroneous reasoning that may hide behind high confidence. We design an asymmetric soft reward matrix for verification trajectories (the precise reward formulation is detailed in Appendix F.3 ). By assigning a heavier penalty to false positives (FP), the verifier is shaped into a more stringent screening mechanism. The above rewards are normalized at the prompt level to obtain the final verifier loss \mathcal{L}_{second} , which is added to the first-stage policy loss \mathcal{L}_{first} to update \theta , forming a closed training loop of generator exploration and verifier filtering. 5 Experiments 5.1 Set up Detailed experiment setup (model families, datasets, and evaluation metrics) is in Appendix D . Baselines We compare with three existing test-time reinforcement learning methods. (1) TTRL ( Zuo et al. 2025 ) , which samples multiple trajectories from unlabeled data and leverages the majority vote as pseudo-labels to formulate reward signals; (2) SCRL ( Yan et al. 2026 ) , a consensus-driven approach that reinforces correct reasoning paths with positive pseudo-labels when strong agreement is reached, while using generative uncertainty to construct negative pseudo-labels for pruning incorrect paths otherwise; and (3) Co-Rewarding-III ( Zhang et al. 2025a ) , which enhances reward stability via multi-view cross-validation. The latter assigns positive rewards only if the model yields consistent answers across semantically rewritten problem variants, while additionally employing an Exponential Moving Average (EMA) teacher to provide reference pseudo-labels for policy updates. Table 1 : Main experimental results on six reasoning benchmarks. Each cell reports pass@1 / pass@16 . Ours refers to our proposed TTRL-CoCoV. \Delta rows show gains of Ours over TTRL: absolute (top) and relative % (bottom). † Uses labeled reward data; all other methods are label-free . Method AIME24 MATH500 AIME25 AMC GPQA DAPO Avg [][] Qwen3-4B-Base Base 10.4 / 35.5 67.5 / 90.2 7.8 / 31.6 39.0 / 73.3 34.2 / 83.4 28.7 / 65.5 31.3 / 63.3 GRPO † 22.0 / 45.5 81.6 / 91.9 22.5 / 42.5 57.1 / 81.0 40.5 / 83.4 49.9 / 77.0 45.6 / 70.2 TTRL 10.8 / 14.9 72.8 / 85.3 3.3 / 17.6 42.2 / 60.0 35.6 / 74.9 31.2 / 55.1 32.7 / 51.3 SCRL 12.1 / 25.0 75.3 / 88.1 10.6 / 25.4 46.8 / 67.6 34.4 / 75.8 34.7 / 60.6 35.7 / 57.1 Co-rewarding 15.9 / 26.1 77.5 / 86.2 7.7 / 19.9 49.6 / 64.5 35.7 / 68.7 40.1 / 60.1 37.8 / 54.3 [][] Ours 20.1 / 39.0 81.6 / 93.2 18.2 / 47.0 53.7 / 80.8 38.1 / 84.0 43.0 / 76.0 42.5 / 70.0 [][] \boldsymbol{\Delta} (Ours – TTRL) +9.3 / +24.1 +86% / +161% +8.8 / +7.9 +12% / +9% +14.9 / +29.4 +451% / +167% +11.5 / +20.8 +27% / +34% +2.5 / +9.1 +7% / +12% +11.8 / +20.9 +37% / +37% +9.8 / +18.7 +30% / +36% [][] Qwen3-8B-Base Base 11.0 / 35.4 63.3 / 91.2 9.1 / 31.5 38.5 / 79.6 35.3 / 88.9 29.3 / 69.9 31.1 / 66.1 GRPO † 26.5 / 55.3 85.8 / 94.9 22.3 / 38.8 65.1 / 86.3 48.1 / 88.3 55.7 / 81.5 50.6 / 74.2 TTRL 13.6 / 28.4 77.0 / 87.7 9.9 / 24.3 50.6 / 72.9 39.1 / 79.6 38.7 / 64.3 38.2 / 59.5 SCRL 13.5 / 32.6 77.8 / 87.9 10.7 / 26.4 51.1 / 71.2 38.2 / 82.5 38.4 / 63.0 38.3 / 60.6 Co-rewarding 15.9 / 30.5 80.5 / 89.5 12.5 / 21.0 54.9 / 74.3 41.5 / 75.6 44.7 / 63.9 41.7 / 59.1 [][] Ours 22.0 / 50.5 82.5 / 94.6 16.3 / 37.9 56.4 / 84.3 43.6 / 89.9 45.3 / 77.6 44.4 / 72.5 [][] \boldsymbol{\Delta} (Ours – TTRL) +8.4 / +22.1 +61% / +77% +5.5 / +6.9 +7% / +7% +6.4 / +13.6 +64% / +56% +5.8 / +11.4 +11% / +15% +4.5 / +10.3 +11% / +12% +6.6 / +13.3 +17% / +20% +6.2 / +13.0 +16% / +21% 5.2 Experimental Results TTRL-CoCoV significantly outperforms all label-free baselines across pass@1 and pass@16 metrics while mitigating the exploration degradation inherent in standard TTRL. As shown in Table 5.1 , standard TTRL often suffers from late-stage mode collapse, causing its pass@16 performance to sharply fall behind the Base model (e.g., dropping from 31.6% to 17.6% on AIME25 for Qwen3-4B). In contrast, TTRL-CoCoV sustains diverse reasoning exploration, effectively preventing such performance drops. Averaged across all six benchmarks, our method yields absolute gains of +9.8% and +18.7% in pass@1 and pass@16 over TTRL on the Qwen3-4B architecture, alongside robust improvements of +6.2% and +13.0% on the larger Qwen3-8B. Figure 3: Training dynamics of Verifier: validation correct rate increases while error rate declines. TTRL-CoCoV fosters a synergistic enhancement of both generative and verification capabilities. Tracking the verifier’s internal metrics during training (Fig. 3 ) reveals a steady upward trend in verification accuracy, with a consistently converging verification error rate. These dynamics validate the efficacy of our design: the generator produces high-quality reasoning trajectories, while the inherently refined verifier filters out false consensus. Ultimately, this joint parameter update establishes a powerful virtuous cycle of mutual improvement. TTRL-CoCoV approaches or even exceeds the supervised upper bound set by ground-truth labels. Although TTRL-CoCoV is a purely label-free fine-tuning paradigm, its performance closely approaches the upper bound of fully supervised training with ground-truth labels (GRPO). For instance, on the Qwen3-4B architecture, TTRL-CoCoV achieves an average Pass@16 of 70.0% across six benchmarks, nearly matching the GRPO upper bound of 70.2%. More remarkably, it surpasses this supervised ceiling on highly complex tasks, attaining a 47.0% Pass@16 on AIME25 (vs. GRPO’s 42.5%) and matching GRPO’s 81.6% Pass@1 on MATH500. This substantially bridges the gap between label-free and label-supervised learning. Table 2: Pass@1 performance comparison with policy annealing. ∗ denotes our method further optimized with annealing mechanism. Method AIME24 MATH500 GPQA Avg GRPO 22.0 81.6 40.5 48.0 Ours 20.1 81.6 38.1 46.6 [][] Ours ∗ 23.4 82.6 45.5 50.5 [][] \Delta (Ours ∗ - GRPO) +1.4 +1.0 +5.0 +2.5 Post-TTRL-CoCoV policy annealing triggers deep exploitation to surpass performance upper bounds. To investigate whether the model can achieve further convergence after accumulating sufficient exploratory experience, we introduce a two-stage "exploration-to-exploitation" policy annealing mechanism. Through TTRL-CoCoV training, the model constructs a highly diverse pool of correct reasoning trajectories. In the subsequent training phase, we remove the diversity reward and strictly transition the optimization objective toward a single-path Pass@1 greedy maximization (i.e., the exploitation phase). Empirical results demonstrate that this brief convergence phase triggers a secondary surge in model capability (Table 2 ). Not only does it overcome the performance bottleneck of the initial exploratory stage, but it also successfully surpasses the Pass@1 performance of the fully-supervised GRPO across multiple benchmarks. By adopting this "explore-then-exploit" annealing paradigm, our approach effectively bridges the gap between label-free and label-supervised learning, ultimately exceeding the established supervised performance upper bounds. Figure 4: Training and internal verification dynamics of TTRL-CoCoV. (Left & Middle): Reward Accuracy and Label Accuracy. While standard TTRL suffers from late-stage pseudo-label collapse and confirmation bias, TTRL-CoCoV maintains highly stable reward accuracy ( >0.8 ) and smoothly increasing label accuracy. (Right): Validation Correction Rate demonstrating co-evolution. Under joint updates, the correct verification rate steadily climbs, whereas a static verifier suffers from capability mismatch and degrades. 6 Analysis and Discussions Q1: Can TTRL-CoCoV consistently provide high-quality training signals? To investigate whether TTRL-CoCoV can overcome the common pseudo-label noise problem in label-free learning, we track the reward accuracy and pseudo-label accuracy throughout the training process (see Fig. 4 , left and middle). We demonstrate that our confidence-conditioned self-classification and self-verification mechanisms effectively break the vicious cycle of confirmation bias. By accurately filtering out false high-consensus noise, TTRL-CoCoV continuously distills high-quality training signals and completely avoids the late-stage reward collapse seen in traditional TTRL (a detailed numerical analysis is provided in Appendix E.2 ). Q2: How does the co-evolution mechanism in TTRL-CoCoV affect training? To investigate the interaction between generation and verification capabilities under shared weights, we established an ablation baseline with a frozen verifier and analyzed the internal verification dynamics (see Fig. 4 , right). We demonstrate that without co-evolution, a static verifier suffers from a severe capability disconnect as the generator evolves, ultimately becoming a performance bottleneck. Conversely, joint updates create a virtuous cycle: the generator’s output continuously calibrates the verifier’s discriminative intuition, while the upgraded verifier improves reward accuracy for the generator. This achieves true co-evolution and prevents the significant degradation in downstream exploration capabilities caused by a static verifier (detailed downstream task performance and numerical error metrics are provided in Appendix E.3 ). Q3: Why does the length-diversity reward in TTRL-CoCoV enhance the model’s exploration capability? In traditional TTRL, when the model achieves high consistency on a particular problem, it tends to exploit shortcut learning by outputting short, templated answers to quickly obtain rewards. Our analysis reveals that without the length diversity reward ( R_{div} ), the generated responses become highly homogenized. By introducing R_{div} , we successfully force the model out of this comfort zone. Rewarding correct trajectories that deviate from the mean length effectively stimulates the model’s exploration capability, enabling it to explore different problem-solving approaches and generate more diverse chains of thought while maintaining high accuracy (detailed length dynamics and variance curves are provided in Appendix E.4 ). Q4: How do different model sizes and types affect the effectiveness of TTRL-CoCoV? To comprehensively evaluate the scalability and generalization of our proposed method, we conduct comparative experiments across varying parameter scales and diverse model architectures. We demonstrate that our framework aligns with reinforcement learning scaling laws, with performance gains increasing significantly as model size grows due to the larger models’ stronger inherent reasoning and verification capabilities. Furthermore, cross-model evaluations reveal that while our method achieves stable improvements across different architectures, the magnitude of these gains is highly dependent on domain knowledge: base models with richer prior expertise in the target domain unlock substantially higher exploration and verification gains (comprehensive benchmark evaluations and cross-model comparisons are detailed in Appendices E.5 and E.6 ). Q5: Why TTRL-CoCoV uses an asymmetric soft penalty matrix? To validate the design of our asymmetric soft penalty reward matrix during verification, we evaluate its noise-filtering capability against a standard symmetric reward baseline. We demonstrate that while a symmetric setting allows persistently elevated false positive ratios that corrupt gradient signals, our asymmetric strategy—assigning a higher penalty weight to false positives—imposes a rigorous screening criterion. This strictly mitigates the impact of erroneous gradients on the first-stage policy, effectively enhancing overall reward accuracy in label-free learning (detailed verification dynamics and false positive ratio comparisons are provided in Appendix E.7 ). 7 Conclusion We presented TTRL-CoCoV to address the previously under-explored challenge of Pass@k optimization in unsupervised TTRL. By combining an adaptive co-evolutionary verification mechanism with an exploration-enhancing reward, the framework simultaneously resolves pseudo-label unreliability and trajectory diversity collapse. TTRL-CoCoV not only yields consistent gains in generation coverage, but also leverages the better exploratory capability to drive further exploitation, ultimately surpassing the performance ceiling of fully supervised baselines on challenging benchmarks. Appendix Appendix A Related Work Label-free test-time reinforcement learning. STaR ( Zelikman et al. 2022 ) and SRLMs ( Yuan et al. 2024 ) established the foundation of annotation-free self-improvement. Building on this, TTRL ( Zuo et al. 2025 ) formalized majority-vote consensus over self-sampled rollouts as a general unsupervised fine-tuning paradigm. Subsequent works further strengthened this framework along the dimension of label reliability: SRT ( Shafayat et al. 2025 ) via consistency-based filtering, DARE ( Du et al. 2026 ) via distributional reward estimation, RESTRAIN ( Yu et al. 2025a ) via self-penalization, SCRL ( Yan et al. 2026 ) via complementary positive-negative labeling, and Co-rewarding ( Zhang et al. 2025a ) via cross-view supervision. Our work complements this line by introducing confidence stratification to jointly address pseudo-label quality and trajectory diversity within a unified framework. Synergistic generation and verification. A related line of research uses the asymmetry that verification is easier than generation to construct reliable training signals without external annotation. Early verifier-based methods ( ( Lightman et al. 2023 ) , ( Weng et al. 2023 ) ) relied on dedicated reward models, while more recent work unifies generation and verification within a single model through self-play, adversarial games, or cooperative RL objectives ( ( Zhao et al. 2025 ) , ( Huang et al. 2025 ) , ( Chen et al. 2025b ) , ( Zhang et al. 2025b ) ). Self-Harmony ( Wang et al. 2025a ) , Incentivizing LLMs to Self-Verify ( Zhang et al. 2025c ) , and Learning to Self-Verify ( Chen et al. 2026 ) further investigate conditions under which self-verification yields reliable signals, focusing on training stability, distribution alignment, and over-verification suppression, respectively. CoVerRL ( Pan et al. 2026 ) represents a particularly relevant instantiation, bootstrapping both roles through shared weights in a co-evolutionary TTRL loop. Our work extends this direction by introducing confidence-conditioned verification allocation. Exploration and diversity in LLM reasoning. In standard training-time RL, methods such as Outcome-based Exploration ( Song et al. 2025 ) , PKPO ( Walder and Karkhanis 2025 ) , Pass@k Training ( Chen et al. 2025a ) , and DSDR ( Wan et al. 2026 ) promote diverse solution strategies with access to ground-truth labels. At pure inference time, FoT ( Bi et al. 2024 ) and SPREAD ( Khanh et al. 2026 ) improve output diversity without parameter updates. In the label-free test-time RL setting, SPINE ( Wu et al. 2025 ) and Evol-RL ( Zhou et al. 2025 ) explore diversity-aware objectives, representing early steps toward sustained exploration under unsupervised conditions. Our work builds on this direction by explicitly studying diversity collapse as a distinct failure mode and proposing a confidence-conditioned remedy. Appendix B Algorithmic Pseudocode of TTRL-CoCoV This section presents the algorithmic pseudocode for the TTRL-CoCoV training loop, detailing the conditional verification process and trajectory generation based on majority confidence. Algorithm 1 TTRL-CoCoV Training Loop 1: Pretrained model \pi_{\theta} , dataset \mathcal{D} , thresholds \tau_{high},\tau_{low} , candidate size K , verification attempts m 2: for each batch (x) in \mathcal{D} do 3: Generate N reasoning trajectories \{y_{1},\dots,y_{N}\}\sim\pi_{\theta}(\cdot|x) 4: Compute majority confidence C_{maj} and extract Top- K candidates \{o_{k}\}_{k=1}^{K} 5: if C_{maj}\geq\tau_{high} then \triangleright High-Consistency 6: \mathbb{I}_{verify}\leftarrow 1;y^{*}\leftarrow y_{maj} 7: Calculate \mathcal{L}_{first} with Length-Diversity Reward R_{div} 8: else if \tau_{low}\leq C_{maj}<\tau_{high} then \triangleright Mid-Consistency 9: \mathbb{I}_{verify}\leftarrow 0;y^{*}\leftarrow y_{maj} 10: Calculate baseline \mathcal{L}_{first} (No verification) 11: else \triangleright Low-Consistency 12: \mathbb{I}_{verify}\leftarrow 1 ; Compute \mathrm{VPR}(o_{k}) for Top- K via verifier 13: if \exists o_{k} with \mathrm{VPR}>0.5 then 14: y^{*}\leftarrow best candidate in true_set 15: else 16: \mathcal{L}_{total}\leftarrow 0 ; Continue \triangleright Skip update 17: end if 18: end if 19: if \mathbb{I}_{verify}==1 then 20: Generate m verification trajectories per candidate 21: Compute \mathcal{L}_{second} using Asymmetric Rewards matrix 22: end if 23: \mathcal{L}_{total}=\mathcal{L}_{first}+\mathbb{I}_{verify}\cdot\mathcal{C}_{pseudo}\cdot\mathrm{Mean}(\mathcal{L}_{second}) 24: Update parameters \theta using gradients \nabla_{\theta}\mathcal{L}_{total} 25: end for Appendix C Implementation Details C.1 RL Hyperparameters The specific hyperparameters utilized during the reinforcement learning phase of TTRL-CoCoV are outlined in Table 3 . This includes generator, verifier, and PPO-specific configurations. Table 3: TTRL-CoCoV Training Settings Method Hyperparameters Generator n_{\text{vote}}=64 n_{\text{samples\_per\_prompt}}=32 Top- p = 1.0 Training Temperature = 1.0 K_{pass}=4 Verifier Temperature: T_{high}=1.0 , T_{low}=0.6 \tau_{high}=0.6 , \tau_{low}=0.4 Top- K candidates: K_{high}=3 , K_{low}=5 Top- p = 0.85 n_{\text{verification\_samples}}=8 Length Diversity \lambda_{\text{div}}=0.05 , C_{max}=2 \tau_{high}=0.6 PPO Trainer Learning Rate = 5\times 10^{-7} \gamma=1.0 , \lambda=1.0 Use KL Loss = True KL Coefficient = 0.001 Batch Sizes \text{train\_batch\_size}=32 \text{rollout\_batch\_size}=32 \text{mini\_batch\_size}=1 \text{micro\_batch\_size}=8 Lengths Prompt Max Length = 1024 Generate Max Length = 4096 Verify Max Length = 2048 Training Schedule Epochs = 1 Evaluation n_{\text{samples}}=32 Temperature T=0.6 Top- p=0.95 , Top- K=20 Metrics: Pass@1, Pass@16 C.2 Hardware Setup All experiments, including the reinforcement learning training phase and subsequent evaluations of the TTRL-CoCoV framework, were conducted on a compute node equipped with 4 \times NVIDIA H100 GPUs. C.3 Self Verification Prompt Box B.2: Self Verification Prompt for Mathematical Hypothesis Testing Problem: {problem} [Hypothesis to Test] A previous attempt at this problem resulted in the following answer: {candidate_answer} [Task] Act as a rigorous mathematical reviewer. Treat the previous answer ( {candidate_answer} ) as a given hypothesis. Plug this answer BACK into the original problem conditions. Perform a rigorous backward-substitution to check if it satisfies all constraints or if it leads to a mathematical contradiction. You MUST conclude your verification by explicitly stating either "Verification Result: True" (if the hypothesis perfectly satisfies all conditions) or "Verification Result: False" (if it leads to any contradiction). You MUST strictly use the following XML format for your response: (Your step-by-step backward substitution checking if {candidate_answer} contradicts the problem conditions) Verification Result: [True/False] Appendix D Additional Experimental Setup This section provides comprehensive details regarding our experimental configurations to ensure full reproducibility. Specifically, we outline the diverse suite of foundational models, the training and evaluation datasets, and the exact sampling procedures used for our metrics. Models For our main experiments, we evaluate the effectiveness of our method using Qwen3-4B-Base and Qwen3-8B-Base ( Yang et al. 2025 ) as the foundational models. To further assess scalability and generalizability across varying model capacities and architectures, we conduct auxiliary experiments and ablation studies on a diverse suite of models. These include lightweight variants (Qwen3-0.6B and 1.7B Base models ( Yang et al. 2025 ) ), a domain-specific model (Qwen2.5-Math-7B ( Yang et al. 2024 ) ), and OctoThinker-8B-Hybrid-Base ( Wang et al. 2025b ) , which is continually pre-trained from Llama3.1-8B ( Grattafiori et al. 2024 ) . Datasets We train our model using DAPO-14k-MATH, an English subset derived from DAPO-Math-17k ( Yu et al. 2025b ) , comprising 14,000 deduplicated and standardized math reasoning samples with varying difficulty levels. To evaluate performance, we conduct experiments on six widely-recognized benchmarks, including four specialized mathematical reasoning tasks: MATH-500 ( Hendrycks et al. 2021 ) , AIME24, AIME25, and AMC ( Li et al. 2024 ) , as well as two benchmarks for broader capabilities: GPQA-Diamond ( Rein et al. 2023 ) and the DAPO-14k-MATH-test set ( Yu et al. 2025b ) . Metrics For each problem, we independently sample 32 reasoning trajectories with a temperature of 0.6 and a top- p of 0.95, and we compute the following metrics ( Zhou et al. 2025 ) : • pass@1 : The average accuracy across the 32 sampled trajectories. • pass@16 : The probability of obtaining at least one correct answer when randomly sampling 16 trajectories (with replacement) from the total of 32, averaged over 1,000 bootstrap iterations. Appendix E Additional Experimental Results Table 4: Pass@1 vs. Pass@k performance. Standard TTRL severely collapses response diversity (Pass@k), and naively adapting the Pass@k objective (Pass@k TTA) fails to fully recover it due to unmitigated pseudo-label noise. Method AIME24 AIME25 AMC Qwen3-4B-Base 10.4 / 35.5 7.8 / 31.6 39.0 / 73.3 TTRL 10.8 / 14.9 3.3 / 17.6 42.2 / 60.0 Pass@k TTA 11.6 / 27.0 8.5 / 24.6 47.1 / 72.1 E.1 Detailed Empirical Analysis of Naive Pass@k Adaptation To empirically demonstrate the failure of naively adapting the Pass@k objective in label-free settings, we evaluated a straightforward adaptation (denoted as Pass@k TTA), which treats non-majority candidate responses as N_{\text{neg}} , using the Qwen3-4B-Base model across the AIME24, AIME25, and AMC benchmarks. Main results are present in Table 4 . Our observations reveal that while standard TTRL generally improves single-pass accuracy (Pass@1), it triggers a severe collapse in generation diversity (Pass@k). For instance, on the AIME24 benchmark, the Pass@k metric drops drastically from the base model’s 35.5 to 14.9 after standard TTRL training. Furthermore, implementing Pass@k TTA only partially mitigates this degradation—recovering the Pass@k to 27.0 on AIME24—which still falls significantly short of the base model’s original exploratory capacity. Similar performance trends were consistently observed on the AIME25 and AMC datasets. These results empirically confirm our core conclusion: without a ground-truth oracle to guarantee label correctness, naively rewarding low-consistency exploratory signals is fundamentally compromised by severe pseudo-label noise, ultimately failing to sustain the required exploratory space for complex reasoning tasks. E.2 Detailed Numerical Analysis of Noise Resilience To further elaborate on the training dynamics presented in Fig. 4 (left and middle) of the main text, we provide a detailed numerical breakdown of the accuracy trends. In traditional TTRL, the model is prone to reward collapse in the mid-to-late training stages. The underlying reason is often the overconfidence of the policy: as optimization progresses, the model tends to prematurely converge to erroneous consensus on difficult samples, distorting low-consistency guesses into false high-consensus errors, which causes the reward accuracy to drop sharply to around 0.4. In contrast, our empirical results show that even in the later stages where baseline methods completely collapse, TTRL-CoCoV maintains its reward accuracy robustly at a high level of 0.8. In terms of label accuracy, our method establishes an early lead of approximately 0.1. As the two roles of generator and verifier co-evolve, this gap further expands to 0.2 in the later stages. Ultimately, our label accuracy converges smoothly to a high level around 0.7. These detailed results demonstrate that TTRL-CoCoV can accurately identify and filter out false high-consensus noise throughout the entire cycle. E.3 Detailed Impact of Generator-Verifier Co-evolution To provide further empirical evidence for the necessity of joint updates discussed in the main text, we detail the downstream task performance and specific numerical changes in verification error metrics. As shown in Fig. 5 , the absence of the co-evolution mechanism leads to a significant degradation in downstream task performance. Specifically, on the highly challenging AIME25 benchmark, Pass@1 drops from 18.2% to 9.8%, while Pass@16 plummets from 47.0% to 28.8%. These results intuitively demonstrate that the co-evolution of the generator and verifier is crucial for enhancing the model’s exploration capabilities. Regarding the internal verification dynamics, while the main text illustrates the steady climb of the validation correct rate under joint updates (refer to Fig. 4 , right), we present the corresponding detailed dynamics of the validation error rate and format error rate in Fig. 6 . Without updating the verifier, the generator’s problem-solving ability continues to improve during fine-tuning, yet the static verifier’s discriminative upper bound remains locked at the level of the base model. This capability mismatch causes the validation error rate to rise above 0.35. In contrast, when co-updating is enabled, region A continuously provides high-quality positive examples, while region B supplies filtered data samples to assist verifier updates. Consequently, the overall validation error rate and the format error rate drop significantly, ultimately converging to a negligible level. Figure 5: Impact of verifier co-evolution on downstream task performance. Freezing the verifier (w/o verifier update) consistently degrades performance across all five benchmarks, while joint generator-verifier updates (w/ verifier update) achieve the best results. Figure 6: Detailed internal verification error dynamics. Under joint updates (w/ verifier update), both the validation error rate (FP+FN) and format error rate drop significantly. Conversely, a static verifier (w/o verifier update) suffers from capability mismatch, leading to an escalating validation error rate. E.4 Detailed Impact of Length Diversity Reward To further illustrate the impact of the length diversity reward ( R_{div} ) on preventing shortcut learning as discussed in the main text, we compare the dynamic curves of response lengths throughout the training process. As shown in Fig. 7 , we observe a significant difference when ablating this mechanism. After removing the length diversity reward, not only does the model’s average response length rapidly drop to around 1000, but the standard deviation of response length also shrinks severely to the range of 600-800 in the later stages of training. This indicates that the generated responses become highly homogenized. In contrast, with the introduction of R_{div} , the model’s average response length not only avoids shortening but actually grows significantly to the 1500-2500 range. Furthermore, after an initial period of fluctuation, the standard deviation of length rises rapidly and remains robustly at a high level of 1200-1400. This dynamic comparison intuitively reveals that R_{div} provides effective bonuses to correct trajectories that deviate from the mean length, ensuring the model maintains high trajectory diversity. Figure 7: Impact of the length-diversity reward ( R_{div} ) on the mean and standard deviation of response lengths. Without the diversity penalty (red), the standard deviation sharply drops to 600-800, indicating severe mode collapse and shortcut learning. With R_{div} enabled (blue), the model sustains robust trajectory diversity (Std 1200-1400) while maintaining correct mathematical intuition. E.5 Detailed Evaluation of Scalability Across Model Sizes To further elaborate on the scalability discussed in the main text, we conduct cross-scale experiments on the Qwen series of models. As shown in Fig. 8 , the experimental results indicate that our method consistently outperforms the baselines on core reasoning tasks such as AIME24 and MATH500. More importantly, it exhibits performance gains that increase significantly with the model size. This trend is consistent with the scaling law in reinforcement learning: larger models possess stronger reasoning and verification capabilities, thereby providing more accurate error-filtering signals to our verification mechanism and yielding greater performance improvements. Figure 8: Scalability of TTRL-CoCoV across model sizes. TTRL-CoCoV yields consistent Pass@1 improvements across all model sizes. For Pass@k, the gains become more pronounced as model size increases (4B and 8B), suggesting that the co-evolution mechanism benefits more from larger model capacity. Performance on smaller models (0.6B and 1.7B) is mixed, with improvements on AIME24 and MATH500 but slight declines on AMC and GPQA. E.6 Detailed Evaluation of Generalization Across Model Architectures To evaluate the generalization capabilities and the impact of the base model’s prior knowledge, we perform cross-model comparisons under similar parameter scales (7B/8B). As shown in Fig. 9 , whether on OctoThinker-8B, a hybrid base model focused on general capabilities, or Qwen2.5-Math-7B, a domain-specific expert model, TTRL-CoCoV achieves stable performance improvements. However, the experiments also reveal a dependency on domain knowledge: our method achieves the most significant gains on math-fine-tuned models when solving complex reasoning tasks, but shows relatively modest improvements on general-domain datasets that cover broad common sense, such as GPQA. This phenomenon indicates that the performance improvement of our method is highly correlated with the base model’s original capability in the target domain. Figure 9: Generalization of TTRL-CoCoV across different models (7B/8B scale). We compare three models: Qwen3-8B-Base, OctoThinker-8B, and Qwen2.5-math-7B on AIME24, MATH500, AMC, and GPQA. Our method consistently improves Pass@1 across all models and benchmarks. For Pass@k, gains are most pronounced on AIME24 and AMC. Despite OctoThinker-8B’s weaker base performance, TTRL-CoCoV brings substantial improvements, confirming the method’s robustness across diverse architectures. E.7 Impact of Asymmetric Reward on False Positive Suppression Figure 10: FP ratio dynamics under symmetric vs. asymmetric reward strategies. To further evaluate the necessity of the asymmetric soft penalty reward matrix discussed in the main text, we compare the verification dynamics against a symmetric reward baseline, focusing specifically on the false positive (FP) ratio. Empirical results (see Fig. 10 ) demonstrate a stark divergence in verifier behavior: under the symmetric setting, the verifier fails to sufficiently discriminate incorrect responses, leading to a persistently elevated false positive ratio that introduces significant noise into the generator’s gradient signals. Conversely, the asymmetric strategy, which assigns a higher penalty weight to false positives, imposes a more rigorous screening criterion. Consequently, the false positive ratio is notably reduced to approximately 0.6. This effectively blocks erroneous gradient updates from corrupting the generator. E.8 Ablation Study To further validate the individual contributions of the proposed components in TTRL-CoCoV, we conduct additional ablation study based on the Section 6 , using Qwen3-4B-Base. Table 5 presents the additional results. Necessity of the Verification-Guided Evolutionary Paradigm We first isolate the fundamental contribution of our dual-role architecture by comparing the w/o R_{div} variant (which retains the internal verifier) against the naive baseline (w/o verifier+ R_{div} ). The dismantling of the verification system leads to a consistent performance deterioration across all reasoning benchmarks. This degradation is particularly pronounced on highly complex datasets where generation uncertainty is pervasive; for instance, Pass@1 on AIME25 drops from 13.5 to 8.5, and AIME24 falls from 13.7 to 11.6. These results empirically substantiate that a generation-only evolutionary process is inherently fragile. Without the verifier acting as an internal safeguard to continuously screen and filter low-confidence samples, the model inevitably falls victim to the consensus trap, internalizing severe pseudo-label noise that fatally destabilizes the policy optimization trajectory. Impact of the Trajectory Diversity Bonus ( R_{div} ) . The absence of R_{div} triggers a dramatic contraction in the model’s generation coverage, severely bottlenecking Pass@16 performance. Specifically, Pass@16 plummets from 47.0 to 32.4 on AIME25, and from 93.2 to 89.8 on MATH500. More critically, this diversity collapse directly impairs the model’s peak exploitation capability, evidenced by the corresponding sharp declines in Pass@1 (e.g., an absolute drop of 6.4 on AIME24 and 8.3 on AMC). This confirms our core hypothesis: R_{div} is indispensable for sustaining a broad exploratory space, which provides the essential foundation for discovering novel, robust reasoning trajectories. Table 5: Ablation study results on Qwen3-4B-Base. Results are reported as Pass@1 / Pass@16 for all datasets. Method AIME24 MATH500 AMIE25 AMC GPQA TTRL-CoCoV 20.1 / 39.0 81.6 / 93.2 18.2 / 47.0 53.7 / 80.8 38.1 / 84.0 w/o verifier+ R_{div} 11.6 / 27.0 75.6 / 86.7 8.5 / 24.6 47.1 / 72.1 36.1 / 75.0 w/o R_{div} 13.7 / 28.6 76.9 / 89.8 13.5 / 32.4 45.4 / 72.2 35.3 / 77.1 Appendix F Detailed formulations F.1 Mathematical Formulation of GRPO In the section 2 , we briefly introduce Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) . Here, we provide the complete mathematical formulations. For a given input \boldsymbol{x} , the policy samples N responses \{\boldsymbol{y}_{i}\}_{i=1}^{N} . Each response receives a reward R_{i} . The sequence-level advantage \hat{A}_{i} is estimated by normalizing the rewards within the sampled group: ^ A _i = R i - mean ({R j } j=1 N ) std ({R j } j=1 N ) .Using this normalized advantage, GRPO optimizes the policy \pi_{\theta} via the standard clipped surrogate objective, formulated as: J ( θ ) = E x ∼ D , y ∼ π θ old [ 1 | y | ∑ t=1 | y | min ( π θ (y t ∣ x , y