Co-Reward: Self-supervised Reinforcement Learning for Large Language Model Reasoning via Contrastive Agreement
Zizhuo Zhang 1 Jianing Zhu 1∗ Xinmu Ge 2,3∗ Zihua Zhao 2∗ Zhanke Zhou 1 Xuan Li 1 Xiao Feng 1 Jiangchao Yao 2† Bo Han 1† 1 TMLR Group, Hong Kong Baptist University 2 CMIC, Shanghai Jiao Tong University 3 Shanghai Innovation Institute · 2025
Read Co-Reward: Self-supervised Reinforcement Learning for Large Language Model Reasoning via Contrastive Agreement by Zizhuo Zhang 1 Jianing Zhu 1∗ Xinmu Ge 2,3∗ Zihua Zhao 2∗ Zhanke Zhou 1 Xuan Li 1 Xiao Feng 1 Jiangchao Yao 2† Bo Han 1† 1 TMLR Group, Hong Kong Baptist University 2 CMIC, Shanghai Jiao Tong University 3 Shanghai Innovation Institute (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00410.txt.
Introduction
Zizhuo Zhang and eight co-authors teach language models to reason without any human-labeled answers, by letting two views of a problem grade each other. Standard reinforcement learning for reasoning leans on checked answers, which are costly to make. Label-free methods that reward a model for agreeing with itself often collapse, settling into confident but empty routines.
Their framework, Co-rewarding, seeks agreement across different views instead. One version pairs each question with a semantically similar one and rewards consistent reasoning across the pair. Another keeps a slow-moving copy of the model as a teacher whose pseudo-labels steady training. Combining the two helped further. Across math reasoning benchmarks, training stayed stable and beat other label-free methods, gaining about 3.3 points on average and over 7 points on one smaller model. In some tests it matched or passed training with true answer labels.
Catalog record
| CREATOR | Zizhuo Zhang 1 Jianing Zhu 1∗ Xinmu Ge 2,3∗ Zihua Zhao 2∗ Zhanke Zhou 1 Xuan Li 1 Xiao Feng 1 Jiangchao Yao 2† Bo Han 1† 1 TMLR Group, Hong Kong Baptist University 2 CMIC, Shanghai Jiao Tong University 3 Shanghai Innovation Institute |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00410
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Co-Reward: Self-supervised Reinforcement Learning for Large Language Model Reasoning via Contrastive Agreement Zizhuo Zhang 1 Jianing Zhu 1∗ Xinmu Ge 2,3∗ Zihua Zhao 2∗ Zhanke Zhou 1 Xuan Li 1 Xiao Feng 1 Jiangchao Yao 2† Bo Han 1† 1 TMLR Group, Hong Kong Baptist University 2 CMIC, Shanghai Jiao Tong University 3 Shanghai Innovation Institute Abstract Although reinforcement learning with verifiable rewards (RLVR) shows promise in improving the reasoning ability of large language models (LLMs), the scaling up dilemma remains due to the reliance on human annotated labels especially for complex tasks. Recent alternatives that explore various self-reward signals exhibit the eliciting potential of LLM reasoning, but suffer from the non-negligible collapse issue. Inspired by the success of self-supervised learning, we propose Co-Reward , a novel RL framework that leverages contrastive agreement across semantically analogical questions as a reward basis. Specifically, we construct a similar question for each training sample (without labels) and synthesize their individual surrogate labels through a simple rollout voting, and then the reward is constructed by cross-referring the labels of each question pair to enforce the internal reasoning consistency across analogical inputs. Intuitively, such a self-supervised reward-shaping mechanism increases the difficulty of learning collapse into a trivial solution, and promotes stable reasoning elicitation and improvement through expanding the input sample variants. Empirically, Co-Reward achieves superior performance compared to other self-reward baselines on multiple reasoning benchmarks and LLM series, and reaches or even surpasses ground-truth (GT) labeled reward, with improvements of up to + 6.8 % +6.8\% on MATH500 over GT reward on Llama-3.2-3B-Instruct. Our code is publicly available at https://github.com/tmlr-group/Co-Reward . Figure 1: Performance overview. Left: the superiority of Co-Reward demonstrated by reasoning performance on MATH500 ( Lightman et al. 2024 ) using different base models; Right: model collapse illustration with the significant drop of validation curves on baselines, also without ground truth. 1 Introduction Large language models (LLMs) ( Achiam et al. 2023 ; Dubey et al. 2024 ; Qwen et al. 2025 ) have demonstrated remarkable general-purpose capabilities in a wide range of linguistic tasks ( Hendrycks et al. ) . To enhance reasoning ability in specific domains, reinforcement learning with verifiable rewards (RLVR) ( Shao et al. 2024 ; Yu et al. 2025 ) is developed as an essential paradigm for post-training with externally verifiable signals, such as program execution results ( Luo et al. 2025 ) or mathematical equivalence ( Shao et al. 2024 ) . Although performance improvement is impressive, the reliance on high-quality ground-truth supervision is considered a major obstacle ( Ouyang et al. 2022 ; Bai et al. 2022 ) to scale up, which motivates the emerging exploration of other promising paradigms such as self-evolving ( Wang et al. 2022 ) to refine reasoning behavior using only unlabeled data. One prominent direction in this space involves leveraging self-reward signals ( e.g. , entropy information ( Prabhudesai et al. 2025 ) and certainty ( Zhao et al. 2025b ) ) to reinforce the high-quality response in reasoning tasks ( Lightman et al. 2024 ) . These approaches reduce labeling cost by using surrogate supervision to guide the optimization, which drives the model toward a higher level of intelligence automatically. However, current self-reward signals easily induce training collapse ( Shafayat et al. 2025 ) as shown in Figure 1 , and thus are limited to scale up on larger foundation models. This requires us to rethink the underlying belief of realizing self-evolving without external feedback ( Wei et al. 2022 ; Shafayat et al. 2025 ; Zhao et al. 2025b ) , or more fundamentally, how to characterize latent reasoning capability beyond the empirical passing rate to specific questions. Conceptually, it can be understood as a kind of abstract knowledge that consists of implicit logical information (e.g., elementary arithmetic). Although confident outputs are prone to being the correct answer, it does not guarantee that the model is based on a valid reasoning path. Instead of solely relying on output-side signals representing necessary results, we should also complement the eliciting process with validity confirmation, which is more aligned with the induction paradigm for the symbolic mechanism ( Yang et al. 2025b ) . Naturally, it brings us back to conventional self-supervised learning ( Tsai et al. ; Liu et al. 2021 ) that captures intrinsic representation via input-level variants ( Hendrycks & Dietterich ) , which initiates a key hypothesis, i.e. contrastive agreement , for our intuition on reasoning refinement. In this work, we introduce Co-Reward , a new self-supervised reinforcement learning framework designed to elicit reasoning capability through contrastive agreement. In the high level, Co-Reward constructs structured training signals from input-side question variants (refer to Figure 2 ), and encourages model to capable with valid reasoning ability. Specifically, for a given input, we generate a set of rephrased prompts with equivalent semantic intent. The model’s responses to these analogs are aggregated by majority voting to form a pseudo-consensus answer, which is then used to supervise the original input through a reward signal that reflects the alignment with the dominant reasoning trajectory. Based on the above atom operation, Co-Reward forms a dual-path structure to shaping the reward: each majority-voted answers serve as cross-reference to the paired questions for optimization, which explicitly expand the input coverage for reasoning validity. Using the contrastive agreement as a proxy, Co-Reward encourages the model to reinforce reasoning strategies that are robust to linguistic variation, while suppressing trivial solutions that induce model collapse for fake reward. It provides a principled self-training signal that requires no ground-truth labels, yet elicit an intrinsic reasoning ability that valid for broader input coverage. Empirically, we evaluate Co-Reward on a series of reasoning benchmarks using different base models such as the Llama ( Dubey et al. 2024 ) and Qwen ( Qwen et al. 2025 ) series. It outperforms other self-reward baselines across multiple reasoning benchmarks and LLM series, and achieves performance comparable to or exceeding that of GT labeled reward, with improvements of up to +6.8\% over GT reward on Llama-3.2-3B-Instruct (refer to Table 1 ). In addition, it can also achieve the superior test-time training (refer to Table 2 ) performance due to its independence on ground-truth label. We observe that Co-Reward leads to more stable optimization and mitigates collapse behaviors, and conduct ablation studies to provide a comprehensive understanding. These findings demonstrate the promise of Co-Reward for unleashing the latent reasoning capability. To summarize, our main contributions are as follows, • We introduce a new perspective, from self-supervised learning, to elicit latent reasoning capability via contrastive agreement, which aligns with the induction paradigm from input-side information. • We propose Co-Reward, a novel self-supervised reinforcement learning framework that forms a dual-path to construct self-generate rewards and cross-overly supervise input variants. • We empirically demonstrate the general effectiveness of Co-Reward to achieve superior RL and test-time training performance, and also present various ablation studies and further analyses. 2 Preliminary Problem Setups. Given a large language model \pi_{\theta} parameterized by \theta , and a dataset D consisting of question-answer pairs (x,a) , the model generates a response y\sim\pi_{\theta}(\cdot|x) auto-regressively. In particular, y=(y^{1},y^{2},\cdots,y^{n}) and each token is generated according to the conditional probability y^{k+1}\sim\pi_{\theta}(\cdot|x,y^{<k}) given y^{<k} denotes the first generated k tokens. We