When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO
Yu Li, Tian Lan, Zhengling Qi · 2026
Read When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO by Yu Li, Tian Lan, Zhengling Qi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13134.txt.
Introduction
Yu Li and co-authors sharpen Group Relative Policy Optimization by letting right and wrong answers teach each other. GRPO scores outputs against a group mean, yet treats each sample as if it stood alone. That wastes the natural contrast between good and failed traces in one group. Their bilateral context conditioning lets the model cross-reference both kinds during updates. A reward-confidence correction steadies the advantage baseline as well. Math reasoning tests improved across models and variants.
Catalog record
| CREATOR | Yu Li, Tian Lan, Zhengling Qi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56031 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13134
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO Yu Li, Tian Lan, Zhengling Qi Abstract Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning ( BiCC ), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction ( RCC ) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at https://github.com/Skylanding/BiCC . 1 Introduction Large language models (LLMs) have demonstrated remarkable reasoning capabilities through Reinforcement Learning with Verifiable Rewards (RLVR), where models learn from outcome-based feedback such as binary signals of correctness on the mathematical problem solving ( 13 ; 25 ; 7 ) . Among various policy optimization approaches, Group Relative Policy Optimization (GRPO) has emerged as a prevalent method due to its simplicity: rather than learning a separate critic as in Proximal Policy Optimization ( 24 , PPO;) , GRPO samples multiple candidate solutions for each query and uses their relative performance to estimate advantages ( 7 ) . The group-based design has proven highly effective for training reasoning models and has been adopted in several state-of-the-art methods ( 16 ; 34 , e.g.,) . However, vanilla GRPO misses on an important opportunity. Consider a group eight solution samples partitioned into correct and incorrect subsets. The two subsets often exhibit distinct reasoning patterns, as correct solutions tend to share successful strategies while incorrect ones reveal common failure modes ( 28 ; 27 ) . Yet GRPO fails to exploit this partitioned structure ( 36 ) . By evaluating each solution in isolation with respect to only group mean, the algorithm remains ’blind’ to the rest of the group except for calculating advantages. Consequently, contrastive signals that could be derived from comparing successful and failed paths within the same context has not been leveraged To this end, we formalize the above observation by first mathematically reformulating the GRPO objective into a contrastive form. The reformulation reveals that GRPO implicitly maximizes the margin between average policy ratios of correct and incorrect samples. Such reformulation makes the important partition structure behind GRPO visible and exploitable. Building on the insight, we propose Bilateral Context Conditioning ( BiCC ) , which enables explicit cross-partition information flow. The key idea is to let the model observe incorrect attempts when evaluating correct ones, and vice versa, strengthening the signal within the same group to update the policy. Grounded in Learning Using Privileged Information ( 19 , LUPI;) , the opposite-partition samples serve as privileged context available only during training, enabling contrastive learning with zero inference overhead. To stabilize training under bilateral conditioning, we further introduce Reward-Confidence Correction ( RCC ) . Through the first-order approximation of the variance-minimizing baseline under importance sampling method, we derive a covariance-based correction term related to reward to further reduce the variance in GRPO. It accounts for the correlation between the model’s output confidence and the resulting reward, effectively reducing gradient variance without using additional sampling or auxiliary models. The proposed two mechanisms operate on complementary components of the GRPO and its variants: BiCC modifies the policy ratio by incorporating cross-partition context, while RCC refines the advantage estimation through baseline correction. We evaluate on Qwen3-4B and Phi-4-mini across four mathematical reasoning benchmarks: Math500, AMC 2023, AIME 2024, and AIME 2025. BiCC yields consistent gain of 0.3–1.9 percentage points across settings, with larger gains on weaker base models. RCC further stabilizes training by reducing gradient variance by 25–35%. Our contributions are as follows: • A contrastive reformulation of GRPO that exposes implicit partition structure within sampled groups, revealing that the objective maximizes the margin between policy ratios of correct and incorrect samples. • Bilateral Context Conditioning ( BiCC ), which enables right and wrong attempts to inform each other through cross-partition context augmentation, grounded in LUPI. • Reward-Confidence Correction ( RCC ), derived from first-order approximation of the optimal baseline, which reduces gradient variances and stabilizes BiCC training by incorporating reward-confidence covariance. • Empirical validation across GRPO and its variants on two base models, demonstrating consistent gains on competition-level mathematical reasoning benchmarks. 2 Preliminary 2.1 Reinforcement Learning for LLM reasoning Consider a query q drawn from a task distribution \mathcal{D} and an output sequence o=(o_{1},\ldots,o_{T}) with T tokens. The policy \pi_{\theta} , parameterized by LLM weights \theta , defines a distribution over outputs given q that factorizes autoregressively: \pi_{\theta}(o|q)=\prod_{t=1}^{T}\pi_{\theta}(o_{t}|q,o_{<t}) (1) where o_{<t}=(o_{1},\ldots,o_{t-1}) denotes the prefix. For reasoning tasks with verifiable answers, the reward function R(o,q)\in\{0,1\} is binary, where 1 is referred to as correct and 0 otherwise. The objective for training a reasoning model is to maximize expected reward over the policy weight \theta : J(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta}(\cdot|q)}[R(o,q)] (2) PPO. Proximal Policy Optimization ( 24 ) updates the policy using importance sampling from a previous iterate \pi_{\theta_{\text{old}}} . At state s_{t}=(q,o_{<t}) with action a_{t}=o_{t} , the policy importance ratio is \rho_{t}=\pi_{\theta}(a_{t}|s_{t})/\pi_{\theta_{\text{old}}}(a_{t}|s_{t}) . The clipped surrogate objective as an approximation to ( 2 ) is defined as L^{\text{CLIP}}(\theta)=\mathbb{E}\bigl[\min\bigl(\rho_{t}A_{t},\text{clip}(\rho_{t},1{-}\epsilon,1{+}\epsilon)A_{t}\bigr)\bigr], (3) where \epsilon is the clipping threshold. The advantage A_{t}=Q(s_{t},a_{t})-V(s_{t}) measures how much better action a_{t} is compared to average, with Q and V denoting action-value and state-value functions estimated by a learned critic. GRPO. Group Relative Policy Optimization ( 7 ) eliminates the critic model by estimating advantages from grouped samples. For each query q , GRPO samples G outputs \{o_{1},\ldots,o_{G}\} from \pi_{\theta_{\text{old}}} , where o_{i}=(o_{i,1},\ldots,o_{i,T_{i}}) has T_{i} tokens and receives reward r_{i}=R(o_{i},q) . The token-level objective of GRPO is defined as J_{\text{GRPO}}(\theta)=\mathbb{E}_{q\sim\mathcal{D}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_{i}}\sum_{t=1}^{T_{i}}L_{i,t}^{\text{CLIP}}\right] (4) where L_{i,t}^{\text{CLIP}}=\min\bigl(\rho_{i,t}A_{i},\text{clip}(\rho_{i,t},1{-}\epsilon,1{+}\epsilon)A_{i}\bigr) and \rho_{i,t}=\pi_{\theta}(o_{i,t}|q,o_{i,<t})/\pi_{\theta_{\text{old}}}(o_{i,t}|q,o_{i,<t}) . The advantage A_{i} is constant across all tokens within o_{i} and computed via group-wise standardization. Specifically, A_{i}=\frac{r_{i}-\mu}{\sigma},\quad\mu=\frac{1}{G}\sum_{j=1}^{G}r_{j},\quad\sigma=\sqrt{\frac{1}{G}\sum_{j=1}^{G}(r_{j}-\mu)^{2}} (5) The group mean \mu replaces the learned critic in PPO as a baseline, while division by \sigma normalizes the advantage scale across queries due to varying difficulty. 2.2 Policy Gradient with Variance Reduction Once the objective function is constructed, policy gradient methods are typically used. The policy gradient theorem ( 32 ; 26 ) expresses the gradient of expected reward as: \displaystyle\nabla_{\theta}J(\theta) \displaystyle=\mathbb{E}_{\pi_{\theta}}\left[R(o,q)\nabla_{\theta}\log\pi_{\theta}(o|q)\right] (6) \displaystyle=\mathbb{E}\left[\frac{\pi_{\theta}(o|q)}{\pi_{\text{ref}}(o|q)}R(o,q)\nabla_{\theta}\log\pi_{\theta}(o|q)\right]. (7) It is known that subtracting a baseline function b properly can reduce variance without changing the gradient, i.e., \nabla_{\theta}J=\mathbb{E}\left[\frac{\pi_{\theta}}{\pi_{\text{ref}}}(R-b)\nabla_{\theta}\log\pi_{\theta}\right] (8) Under importance sampling with weight w=\pi_{\theta}/\pi_{\text{ref}} , minimizing the variance of the objective function with respect to b yields the optimal baseline: b^{*}=\frac{\mathbb{E}[Rw^{2}]}{\mathbb{E}[w^{2}]}. (9) See the derivation in Appendix A.1 . When w\equiv