Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism

Kaixuan Du, Meng Cao, Hang Zhang, Yukun Wang, Xiangzhou Huang, Ni Li · 2026

Read Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism by Kaixuan Du, Meng Cao, Hang Zhang, Yukun Wang, Xiangzhou Huang, Ni Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16223.txt.

Introduction

Du and co-authors improve label-free training for language models. Such methods learn from the model's own majority votes. The catch is that popular wrong answers can trap training. Their Dual Consensus method uses two vote stages. First the model anchors on its dominant replies. Then brief unlearning lets it explore diverse signals. The training target blends both sets by harmonic mean. Across eight benchmarks, Pass@1 beat plain majority vote.

Catalog record

CREATORKaixuan Du, Meng Cao, Hang Zhang, Yukun Wang, Xiangzhou Huang, Ni Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40591
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism Kaixuan Du, Meng Cao, Hang Zhang, Yukun Wang, Xiangzhou Huang, Ni Li Abstract Current label-free RLVR approaches for large language models (LLMs), such as TTRL and Self-reward, have demonstrated effectiveness in improving the performance of LLMs on complex reasoning tasks. However, these methods rely heavily on accurate pseudo-label estimation and converge on spurious yet popular answers, thereby trapping in a dominant mode and limiting further improvements. Building on this, we propose D ual C onsensus R einforcement L earning 1 1 1 Code is available at https://github.com/v0yager33/DualConsensus . (DCRL), a novel self-supervised training method which is capable of generating more reliable learning signals through a two-stage consensus mechanism. The model initially acts as an anchor , producing dominant responses; then it serves as an explorer , generating diverse auxiliary signals via a temporary unlearning process. The final training target is derived from the harmonic mean of these two signal sets. Notably, the process operates entirely without external models or supervision. Across eight benchmarks and diverse domains, DCRL consistently improves Pass@1 over majority vote while yielding more stable training dynamics. These results demonstrate that DCRL establishes a scalable path toward stronger reasoning without labels. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach to boosting the performance of Large Language Models (LLMs), enabling superior reasoning capabilities through long chain-of-thought Wei et al. 2023 reasoning on various challenging benchmarks OpenAI et al. 2024 ; DeepSeek-AI et al. 2025 ; Yang et al. 2025 . However, typically implemented via algorithms such as Group Relative Policy Optimization (GRPO) Shao et al. 2024 , current RLVR approaches heavily rely on human-annotated datasets or, at a minimum, environments that provide verifiable ground-truth signals Le et al. 2022 ; Wang et al. 2024a . This reliance restricts its generalizability to fully unlabeled or distribution-shifted tasks where neither human annotations Ziegler et al. 2020 nor executable environments are accessible. As LLMs approach or surpass human-level performance, they will inevitably operate in domains where even expert humans cannot provide definitive judgments or reliable evaluations, which motivates the exploration of training on unlabeled data. Figure 1: An overview of Dual Consensus Reinforcement Learning (DCRL). Specifically, the policy model assumes two roles: (1) an anchor that generates dominant and reliable responses; (2) an explorer that produces diverse auxiliary signals through a temporary unlearning process. Recent studies Shafayat et al. 2025 ; Zhao et al. 2025a find that LLMs can achieve self-improvement without labeled data. One approach is determinism-based methods Prabhudesai et al. 2025 ; Zhang et al. 2025b , which derive rewards from the confidence of a single policy along trajectories, thereby encouraging low-entropy and high-confidence predictions. These methods can achieve better performance by sharpening the model’s outputs, but it remains debatable whether they are truly effective for improving reasoning capabilities. Another approach is aggregation-based methods Zhang et al. 2025c ; Yu et al. 2025b ; Wu et al. 2025 , which derive rewards from agreement across multiple samples, assuming that cross-sample consistency correlates with correctness. Nevertheless, these approaches still suffer from two critical limitations: • Spurious Reward Signals : Models struggle to generate distinguishable reward signals, especially when tackling hard reasoning tasks; the answers derived from majority vote may themselves suffer from systematic biases Zhao et al. 2025b . In the later stages of training, spurious majority outcomes can come to dominate, yet the correct solutions may instead lie within the minority rollouts. • Lack Exploration Capability : By continuously rewarding consensus across diverse trajectories, the model’s output distribution becomes increasingly rigid and concentrated, resulting in a severe deficiency in exploration capability or even entropy collapse Cui et al. 2025 . Consequently, the model tends to converge to a narrow set of suboptimal responses and exhibits degraded performance when confronted with out-of-domain (OOD) tasks. In this paper, we propose Dual Consensus —a novel framework for Unsupervised Reinforcement Learning with Verifiable Rewards (URLVR) driven by a multi-stage vote mechanism. Our core insight stems from the following intuition: valid reasoning trajectories should not only converge to the dominant mode but also exhibit enhanced robustness when the distribution of the model is artificially flattened. Instead of naively adopting the fragile majority vote as the pseudo-label, we decompose the rollout process into two stages: anchor and explorer . The anchor stage involves normal rollouts where the model generates responses under its current policy, capturing the dominant reasoning mode. Subsequently, in the explorer stage, we introduce a temporary unlearning process to flatten the distribution and enhance exploration, thereby encouraging the generation of diverse auxiliary responses that deviate from the dominant mode. After obtaining the two signal sets, we compute the harmonic mean of their consensus scores to determine the final reward signal. This harmonic mean balances the reliability of the dominant mode (from the anchor stage) and the diversity of potential valid trajectories (from the explorer stage), effectively mitigating the adverse impact of majority vote when it converges to spurious answers. To validate our approach, we demonstrate the effectiveness of Dual Consensus through extensive experiments. We first train models on the large-scale DAPO-14K-Math dataset and evaluate them on multiple established benchmarks. Additionally, we apply test-time adaptation on five distinct datasets to further assess our method. In summary, our key contributions are as follows: • A novel URLVR method, Dual Consensus , which utilizes the intrinsic robustness of the model to guide the model to evolve with policy optimization methods such as GRPO. Notably, the framework is entirely free of external models and enables continuous self-improvement without supervision. • A pseudo-label selection mechanism and reward design that exploits the intrinsic robustness of the model itself to generate more reliable reward signals, mitigating the toxicity of majority vote when it fails. • We empirically verify the general effectiveness of Dual Consensus in boosting LLMs’ reasoning performance via comprehensive experiments, and additionally present systematic ablation studies and in-depth further analyses. 2 Methodology In this section, we present the details of Dual Consensus. It mitigates spurious majority bias by generating diverse signals via Unlearn Then Explore, identifying more accurate labels through Harmonic Election, and stabilizing updates with Adaptive Sampling. Our framework employs Grouped Relative Policy Optimization (GRPO) Shao et al. 2024 as its foundational RL algorithm—it stabilizes training by normalizing advantage estimates across multiple rollouts of the same prompt. For a given input prompt x (paired with its ground-truth label y ), GRPO first samples n rollouts \{y_{i}\}_{i=1}^{n} from the current policy. For each rollout y_{i} , GRPO computes a reward r_{i}=R(y_{i},y\mid x) , then derives the group-normalized advantage \hat{A}_{i} : \hat{A}_{i}=\frac{r_{i}-\bar{r}}{\sigma_{r}} (1) where \bar{r}=\frac{1}{n}\sum_{k=1}^{n}r_{k} denotes the mean reward of the rollout group, and \sigma_{r} is the standard deviation of the group rewards. The GRPO objective optimizes the target policy \pi_{\theta} by maximizing the clipped normalized advantage: \mathcal{L}_{\text{GRPO}}(x,y;\theta)=\\ \frac{1}{n}\sum_{i=1}^{n}\min\bigl(\rho_{i}(\theta)\hat{A}_{i},\;\text{clip}\bigl(\rho_{i}(\theta),1-\epsilon,1+\epsilon\bigr)\hat{A}_{i}\bigr)\\ -\beta\,\mathbb{D}_{\mathrm{KL}}\bigl[\pi_{\theta}(\cdot\mid x)\parallel\pi_{\text{ref}}(\cdot\mid x)\bigr] (2) where \rho_{i}(\theta)=\pi_{\theta}(y_{i}\mid x)/\pi_{\text{old}}(y_{i}\mid x) is the importance sampling ratio. 2.1 Unlearn Then Explore Following the GRPO framework, we first initialize an anchor model (parameterized by \theta^{\prime} ) by cloning the current policy model (parameterized by \theta ), i.e., \theta^{\prime}\leftarrow\theta . We then strategically apply an unlearning strategy Liu et al. 2024 to transform this anchor into an explorer

Read the complete text