\titlefont
JudgeRLVR: Judge First, Generate Second for Efficient Reasoning · 2026
Read \titlefont by JudgeRLVR: Judge First, Generate Second for Efficient Reasoning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08468.txt.
Introduction
JudgeRLVR reorders reasoning training by teaching the model to judge first and generate second. The 2026 arXiv paper addresses reinforcement learning with verifiable rewards, now a standard reasoning paradigm. Training only for final-answer correctness can push models into verbose, aimless exploration. Models then rely on exhaustive trial and error rather than structured planning. Judging first aims to build evaluation skill before generation. If a model can spot a good path early, training may waste fewer tokens on blind search.
Catalog record
| CREATOR | JudgeRLVR: Judge First, Generate Second for Efficient Reasoning |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08468
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
\titlefont JudgeRLVR: Judge First, Generate Second for Efficient Reasoning Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models. However, optimizing solely for final-answer correctness often drives models into aimless, verbose exploration, where they rely on exhaustive trial-and-error tactics rather than structured planning to reach solutions. While heuristic constraints like length penalties can reduce verbosity, they often truncate essential reasoning steps, creating a difficult trade-off between efficiency and verification. In this paper, we argue that discriminative capability is a prerequisite for efficient generation: by learning to distinguish valid solutions, a model can internalize a guidance signal that prunes the search space. We propose JudgeRLVR, a two-stage judge-then-generate paradigm. In the first stage, we train the model to judge solution responses with verifiable answers. In the second stage, we fine-tune the same model with vanilla generating RLVR initialized from the judge. Compared to Vanilla RLVR using the same math-domain training data, JudgeRLVR achieves a better quality–efficiency trade-off for Qwen3-30B-A3B: on in-domain math , it delivers about +3.7 points average accuracy gain with -42% average generation length; on out-of-domain benchmarks, it delivers about +4.5 points average accuracy improvement, demonstrating enhanced generalization. \abscontent 0 0 footnotetext: ⋆ Contribution during internship at Xiaomi LLM-Core Team. 0 0 footnotetext: ⋄ Co-corresponding authors. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted method for improving large language models on complex reasoning tasks ( 32 ; 17 ) . By providing sparse yet automatically checkable supervision signals (e.g., whether the final answer is correct), RLVR enables models to explore strategies beyond supervised demonstrations and has achieved strong results on reasoning benchmarks like math and code. Figure 1 : Qualitative comparison of reasoning patterns between Vanilla RLVR and JudgeRLVR. While outcome-based RL guarantees final answer correctness, it often induces verbose and unstructured exploration (left). JudgeRLVR implicitly regularizes the reasoning process, leading to a direct and coherent solution path (right). Figure 2 : Pipeline for the two-stage training of JudgeRLVR However, RLVR does not merely optimize correctness; it also implicitly shapes a model’s thinking pattern . When trained primarily with a final-answer reward, many LLMs drift toward a generative search style: enumerating tentative branches, revising intermediate steps, and performing explicit self-correction ( 22 ) . Figure 1 shows that the reasoning model’s output can be divided into 2 parts: chain-of-thought (CoT) trajectories and solution responses. CoT trajectories usually include all the details of the model’s lengthy thought process, whereas solution responses tend to present a clear logical process along with the final answer. The resulting CoT trajectories are often long and exhibit frequent backtracking (e.g., heavy use of “but”, “however”, “let’s try again”) ( 3 ) . Long outputs are expensive, but a deeper issue is quality : such trajectories are low in information density. Existing methods such as Kimi K1.5 ( 15 ) and DAPO ( 26 ) introduce length penalties to reduce tokens and stabilize training, but often come with accuracy drops, making it difficult to achieve an ideal quality–efficiency trade-off. This raises a more general question: can we explore a training paradigm that explicitly targets higher-quality thinking patterns, rather than hoping the model eventually “stumbles upon” them through repeated trial-and-error? Cognitive science suggests that expert reasoning is not defined by externalizing exhaustive search, but by early discrimination and pruning : unpromising paths are filtered before they are expanded ( 4 ) . This aligns with common intuition: experts quickly spot low-value reasoning paths and focus on high-yield hypotheses, while novices resort to trial-and-error and frequent backtracking. This intuition motivates the idea that, to achieve efficient reasoning, one should first develop the ability to judge what constitutes direct and valid reasoning, and then learn how to reason for specific problems. Based on the idea above, we propose JudgeRLVR , a two-stage judge-then-generate paradigm. As shown in Figure 2 , in judging stage, we train the model as a judge: given candidate solution responses generated by diverse models, it learns to classify them as correct or incorrect. In generating stage, we initialize the model from the judging stage and apply Vanilla RLVR fine-tuning on the same model. Under this training paradigm, the model learns to implicitly prune low-quality branches prior to generating extensive textual search, thereby fostering more direct and reliable reasoning patterns. It is worth noting that, we do not impose any explicit length or quality penalties on the CoT trajectories; thus, any reduction in verbosity or refinement in reasoning style can be attributed to the discriminative priors transferred from the judging stage. With our training paradigm, a model can achieve an optimal balance between performance and efficiency. Compared to Vanilla RLVR using the same training data, JudgeRLVR achieves a quality–efficiency trade-off on Qwen3-30B-A3B ( 23 ) : on in-domain math , it achieves about +3.7 points average accuracy improvement while reducing average generation length by about 42% ; on out-of-domain benchmarks, it delivers about +4.5 points average accuracy improvement. From extensive experiments, we also analyze the specific impact of this training paradigm on thinking patterns. We observe a shift in the linguistic style of the generated text, with notably fewer explicit backtracking cues such as “but” during reasoning. This suggests that JudgeRLVR moves from an externalized trial-and-error process to an internal decision-making process, thereby producing higher-quality reasoning patterns. In summary, our main contributions are: (i) we propose JudgeRLVR , a new training paradigm for inducing higher-quality thinking patterns via judge-then-generate ; (ii) JudgeRLVR demonstrates a better quality–efficiency trade-off than Vanilla RLVR on both in-domain and out-of-domain tasks; (iii) we provide interpretable linguistic evidence which is consistent with reduced explicit backtracking. 2 Related Work 2.1 RLVR and Verifiable-Reward Training for Reasoning Reinforcement Learning with Verifiable Rewards (RLVR) is a leading approach for improving LLM reasoning using automatic, task-specific rewards, often outperforming supervised fine-tuning on math and logic tasks ( 12 ; 7 ; 25 ) . However, RLVR has notable limitations: it can yield gains even under spurious rewards ( 17 ) , and it mainly reweights existing high-reward reasoning paths rather than inducing new reasoning behaviors ( 27 ) . This is consistent with concerns that RLVR may implicitly encourage suboptimal thinking by optimizing final correctness instead of process quality ( 5 ) . Most variants emphasize reward design or optimization, leaving the quality–efficiency trade-off underexplored. 2.2 Mathematical Reasoning in Large Language Models Mathematical reasoning is a core benchmark for LLMs, where recent progress relies heavily on RLVR and targeted training ( 22 ; 18 ) . Data diversification via paraphrasing can improve generalization ( 18 ) , while extended RLVR training improves accuracy but tends to produce verbose reasoning traces ( 22 ) . Synthetic data scaling further boosts capability yet does not eliminate redundancy ( 31 ) , and scaling studies still report low information density in reasoning outputs ( 20 ) . Overall, improving both correctness and efficiency remains challenging. 2.3 Learning to Judge: Evaluators, Verifiers, and Discriminative Supervision Training evaluators and using discriminative supervision can steer LLM outputs by separating good from bad generations ( 8 ; 21 ) . Discriminative fine-tuning can suppress negative outputs without preference data ( 8 ) , and stronger evaluators improve reasoning-quality assessment via