Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers

Zhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), The Hong Kong University of Science, Technology ETH AI Center, ETH Zurich, University of California, Merced Sun Yat-sen University MBZUAI yangzhch6@gm · 2025

Read Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers by Zhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), The Hong Kong University of Science, Technology ETH AI Center, ETH Zurich, University of California, Merced Sun Yat-sen University MBZUAI yangzhch6@gm (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23152.txt.

Introduction

Picking the best sampled answer fails when the right answer is in the minority. Zhicheng Yang and 6 co-authors train verifiers that critique, not just score. Their Mirror-Critique framework contrasts model solutions with ground-truth ones. A small tuned model writes critique data that says what is wrong and why. Rejection sampling keeps that data high quality.

The critiques cold-start a reinforcement process that sharpens verification skill. At test time, the Mirror-Verifier writes several critiques per candidate. Their aggregate score drives weighted voting or abstention. Accuracy beat majority voting by a clear margin. The solver also grew more honest about its own limits.

Catalog record

CREATORZhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), The Hong Kong University of Science, Technology ETH AI Center, ETH Zurich, University of California, Merced Sun Yat-sen University MBZUAI yangzhch6@gm
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23152

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers Zhicheng Yang Zhijiang Guo Yinya Huang Yongxin Wang, Yiwei Wang Xiaodan Liang Jing Tang, The Hong Kong University of Science, Technology (Guangzhou), The Hong Kong University of Science, Technology ETH AI Center, ETH Zurich, University of California, Merced Sun Yat-sen University MBZUAI [email protected] Project Repo : https://github.com/yangzhch6/Mirror-Critique Abstract Test-time scaling via solution sampling and aggregation has become a key paradigm for improving the reasoning performance of Large Language Models (LLMs). While reward model selection is commonly employed in this approach, it often fails to identify minority-yet-correct answers, which limits its effectiveness beyond that of simple majority voting. We argue that this limitation stems from a lack of informative critique signals during verifier training. To bridge this gap, we introduce Mirror-Critique , a framework that trains a verifier with informative critiques. Our key insight is to leverage the rich critique signal by contrasting model-generated solutions with ground-truth solutions. We deploy a small instruction-tuned model to synthesize high-quality critique data with rejection sampling that teaches the verifier not only what is wrong, but also why. The synthetic data is used to cold-start the LLMs in the RLVR process to further improve the verification ability. The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution, aggregating them into a verify score used for weighted voting or selective abstention. The experimental results show that our Mirror-Verifier significantly outperforms majority voting in terms of solution accuracy and also improves the solver’s honesty to recognize and abstain from answering beyond its capability boundaries. 1 Introduction Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful method for training Large Language Models (LLMs) to perform complex reasoning tasks, enabling significant improvements in domains such as mathematics, code generation, and scientific problem-solving. A common strategy to further boost performance is test-time scaling: generating multiple candidate solutions and aggregating them via methods such as majority voting, verifier model voting, or aggregator model selection. In an ideal scenario, an effective verifier or aggregator should be able to approach Pass@K performance by improving solution accuracy through test-time scaling. However, they often fail to identify minority yet correct solutions, resulting in limitations in their improvement compared to majority voting. This limitation underscores the need for more sophisticated verification mechanisms that can critically evaluate and select solutions. While verifiers have demonstrated promise in detecting flawed reasoning, their training typically depends on binary labels that provide insufficient feedback about why a solution succeeds or fails. This limitation constrains the verifier’s ability to improve its performance meaningfully. One potential approach involves enhancing LLMs with critique capabilities through supervised fine-tuning on critique data. However, obtaining high-quality critique data often requires sampling from closed-source models, making this approach prohibitively expensive. Additionally, the potential of RLVR for training verifiers remains largely unexplored. RLVR could offer significant advantages by improving accuracy while enabling models to recognize their limitations and appropriately abstain from answering questions beyond their capabilities. To this end, we propose Mirror-Critique , a novel framework that synthesizes high quality critiques by contrasting model-generated solutions with ground-truth answers to train a verifier. Our key insight is that such informative critiques can teach the verifier not only to judge correctness, but also to understand the underlying reasoning gaps. We generate high-quality critique data via rejection sampling on an open-source, instruction-tuned model. The synthesized critique data is then used to fine-tune the Base LLMs to address the cold start issue for the RLVR process, further improving the verification ability. The resulting Mirror-Verifier is deployed to generate multiple critiques per solution, which are aggregated into a verification score used for weighted voting or selective abstention during test-time scaling. Extensive experiments on multiple mathematical reasoning benchmarks show that Mirror-Verifier significantly outperforms majority voting and reward-based selection methods, achieving superior accuracy across tasks. Furthermore, it enhances the honesty of the solver–verifier system, enabling it to abstain from questions beyond its capability boundaries, both in test-time scaling and standard (Pass@1) settings. In summary, our main contributions are: • We introduce the Mirror-Critique framework, a novel approach for training verifiers that leverages rich, synthetic critique data generated by contrasting LLM solutions with ground-truth solutions. This synthetic critique data, curated via rejection sampling, provides informative signals that teach the verifier to not only identify errors but also understand their rationale. Unlike other approaches that depend on distillation from larger models, our method is self-contained, synthesizing all training data through internal supervision. • We demonstrate significant accuracy gains in test-time scaling. By using the Mirror-Verifier to aggregate multiple solutions via weighted voting, our approach consistently outperforms strong baselines like majority voting and reward-model selection across multiple mathematical reasoning benchmarks. • We propose the honesty score and show that Mirror-Verifier significantly improves it. This metric quantifies a model’s ability to know what it knows. By abstaining from answers with low verification scores, our framework enhances model honesty while maintaining answer accuracy, reliably recognizing capability boundaries in both test-time scaling and standard (Pass@1) settings. 2 Related Works Reinforcement Learning with Verifiable Reward Reinforcement Learning (RL) has become a standard component in the post-training stage of LLMs. Recent research indicates that RLVR substantially enhances the reasoning performance of LLMs in areas such as mathematics and code generation. A notable advancement was made with OpenAI’s o1 model ( Jaech et al. 2024 ) , which marked a significant leap in reasoning capabilities. This was followed by DeepSeek-R1 ( Guo et al. 2025 ) , where RLVR was shown to activate inherent slow-thinking abilities in a base model—a paradigm now referred to as zero-RL ( Li et al. 2025 ) . Subsequently, multiple Large Reasoning Models (LRMs) have been released, such as Kimi 1.5 ( Team et al. 2025 ) , Gemini-Think ( DeepMind 2024 ) , and QwQ ( Qwen 2024 ) . SimpleRL ( Zeng et al. 2025 ) provided comprehensive empirical studies on zero-RL, while Luo et al. 2025 utilized RLVR to further improve open-source models derived from DeepSeek-R1. A prominent RLVR algorithm adopted in many of these works is GRPO ( Shao et al. 2024 ) . Extending PPO ( Schulman et al. 2017 ) , it achieves notable improvements by evaluating multiple responses to estimate group-relative advantage. GRPO has motivated several variants, including DAPO ( Yu et al. 2025 ) , VAPO ( Yue et al. 2025 ) , and Dr. GRPO ( Liu et al. 2025b ) . Additionally, DARS ( Yang et al. 2025 ) introduces adaptive sampling based on difficulty, leading to gains in both Pass@1 and Pass@K metrics. In this work, we train the verifier while performing zero-RL training of the solver, and further improvements are achieved through the solver-verifier framework in test-time scaling. Test-Time Scaling Test-time scaling through solution sampling and aggregation has become a widely adopted paradigm for improving reasoning performance in LLMs. A common strategy is to use rule-based methods such as majority voting, exemplified by self-consistent decoding ( Wang et al. 2023 ; Brown et al. 2024 ) , which aggregates multiple chain-of-thought trajectories by selecting the most frequent answer. Several lightweight variants have been proposed to enhance this approach, including dynamically adjusting the number of samples or applying heuristic filters (

Read the complete text