Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards Kirill Pavlenko, Alexander Golubev, Simon Karasik, Boris Yangel Abstract Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation , a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts. † † affiliationnotice: Equal contribution 1 Introduction Reinforcement Learning (RL) has gained significant traction as the primary paradigm for post-training Large Language Models (LLMs), enabling them to solve complex tasks that are difficult to supervise with static datasets ( 15 ) . Among recent methods, Group Relative Policy Optimization (GRPO) ( 18 ) and its modifications (DAPO ( 20 ) , GSPO ( 21 ) , CISPO ( 14 ) , ScaleRL ( 11 ) ) have become particularly popular due to their stability and memory efficiency. By normalizing rewards within a sampled group of trajectories rather than relying on a separate value network, these methods avoid the need for maintaining a critic model, significantly reducing the computational footprint of training inherent to classical PPO. Despite these advantages, standard GRPO typically treats each generated completion as a monolithic trajectory: it computes a single scalar advantage and applies it uniformly to all tokens. This formulation is often suboptimal for complex tasks involving sequential subgoals, independent rubrics, or multi-step refinements. Given a sequence of rewards r_{1},...,r_{n} , the standard approach is to construct a scalar function R=f(r_{1},r_{2},...,r_{n}) that maps all rewards into one number and then apply a GRPO-like algorithm ( 17 ; 5 ) . This exacerbates credit assignment and requires careful handcrafting of R to avoid score degradation described in RLCR ( 3 ) , mode collapse, and reward hacking ( 5 ) . Many LLM tasks, however, are naturally segmented : each r_{k} can be assigned to non-overlapping text blocks (e.g., a solution to a problem gets r_{1} , and the following self-reflection block gets r_{2} ). Our core insight is that, in this case, optimization does not have to be framed as a reward design problem. Instead, we can update each block independently based on signals tied specifically to its local objective r_{k} . Motivated by this observation, we introduce Blockwise Advantage Estimation, a family of GRPO-compatible methods that, using only the initial set of rollouts, compute separate proxy-advantages for different text segments and update each segment using only its local objective signal. A central technical challenge is baseline estimation for later segments. Consider a generation partitioned into segments X_{1},X_{2},\ldots , where X_{i} is sampled conditioned on the prefix up to X_{i-1} . To form a low-variance advantage for segment X_{i} , one ideally normalizes the segment-level reward relative to the value of the intermediate state at the end of segment X_{i-1} . A naive solution is to perform Monte Carlo rollouts from the intermediate state, similar to VinePPO ( 10 ) or SPO ( 7 ) , yielding unbiased estimates but incurring prohibitive compute for long-context LLMs and multi-thousand-token generations. We therefore focus on scalable baseline estimators that use only data already produced within the current GRPO group. We study a series of progressively more informative estimators and find that an Outcome-Conditioned Baseline performs best in practice. This allows us to approximate the state-value function and reduce variance without additional inference costs. Our contributions are as follows: • We introduce Blockwise Advantage Estimation , a critic-free, GRPO-compatible framework for multi-objective RL on structured generations that routes objective-specific advantages to the segments of tokens that control them, reducing reliance on hand-tuned scalar reward combinations. • We formalize the conditional baseline problem that arises because later segments are sampled conditioned on earlier sampled segments, and show that the Group Mean Baseline proposed in vanilla GRPO (normalizing rewards for all turns by the within-group mean and variance) can be suboptimal for credit assignment in multi-segment generations. We then propose a compute-efficient Outcome-Conditioned Baseline that approximates boundary-state values using only within-group statistics, reducing variance for later segments without Monte-Carlo rollouts or extra inference. • We demonstrate the efficacy of our method on the task of joint reasoning and uncertainty estimation across three models. Our approach performs on par with a state-of-the-art (SoTA) reward-designed baseline like RLCR in terms of Expected Calibration Error (ECE) and reasoning accuracy. • We demonstrate broader applicability in the additional setting of two-attempt RL for math, and discuss implications for long-horizon agentic tasks where segment-level decoupling enables efficient optimization over extremely long contexts. 2 Related Work Credit assignment beyond trajectory-level advantages. A classical response to coarse trajectory-level credit assignment is to move toward finer-grained advantage estimates. Token-level methods (e.g., PPO with GAE) can in principle provide per-token learning signals, but require fitting a value function, which is empirically difficult in LLM settings due to the diversity of prompt-conditioned states and limited per-prompt data. Recent critic-free approaches instead estimate intermediate advantages by Monte Carlo rollouts from selected intermediate states. VinePPO partitions generations into heuristic steps (e.g., line breaks) and resamples continuations to estimate step-level advantages. Segment Policy Optimization generalizes this idea to arbitrary segment partitions and estimates segment-level advantages via MC. These methods primarily address credit assignment for a single terminal objective and still rely on additional sampling from intermediate prefixes to estimate intermediate values. In contrast, our setting is multi-objective with rewards naturally attached to disjoint text blocks, and our goal is a compute-efficient GRPO-compatible method: to estimate per-block baselines using only the rollouts already present in the group. Stratified and conditional advantage estimation. Several recent works show that advantage normalization can be systematically miscalibrated when trajectories are heterogeneous in ways that affect reward distributions. Stratified GRPO ( 22 ) identifies cross-stratum bias in search agents: using a global baseline across trajectories with different tool-use structures induces an “apples-to-oranges” offset that distorts credit assignment; their Stratified Advantage Normalization computes normalized advantages within homogeneous strata and can be blended with a global estimator for finite-sample stability. CANON ( 1 ) proposes conditional regrouping based on intrinsic rollout metrics (e.g., entropy or response length), computing both inter-group and intra-group advantages. Both lines use trajectory-level stratification to sharpen learning signals for a single reward. Our conditionality problem is different: in segmented generations, later blocks are sampled from different intermediate prefixes inside the same prompt group, so the appropriate baseline for block k is a conditional value \mathbb{E}[r_{k}\mid x,X_{0 is a small constant for numerical stability. GRPO optimizes a PPO-style clipped policy-gradient objective, applying the same scalar advantage \hat{A}^{(i)} to all tokens in the completion: \mathcal{L}_{\text{GRPO}}(\theta)\;=\;-\frac{1}{G}\sum_{i=1}^{G}\sum_{t=1}^{T_{i}}\min\Bigg(\Big(\rho_{i,t}(\theta)\,\hat{A}^{(i)},\\ \;\text{clip}(\rho_{i,t}(\theta),1-\eta,1+\eta)\,\hat{A}^{(i)}\Big)-\beta\mathbb{D}_{KL}[\pi_{\theta}||\pi_{ref}]\Bigg), (1) where \eta is the PPO clip parameter and \rho_{i,t}(\theta)\;=\;\frac{\pi_{\theta}\!\left(y^{(i)}_{t}\mid x,y^{(i)}_{1 , however, the starting state includes the sampled prefix [X_{1};\ldots;X_{k-1}] , which differs across group members. A statistically appropriate advantage for block k should therefore normalize r_{k}^{(i)} relative to the value of the intermediate state at the end of block (k\!-\!1) , i.e., a conditional baseline of the form b_{k}\!\left(x,X_{1 , and apply them to form proxy-advantages \hat{A}_{k}^{(i)}=r_{k}^{(i)}-\hat{b}_{k}^{(i)} (with optional variance normalization as in GRPO): 1. No baseline (raw reward). \hat{b}_{k}^{(i)}=0,\qquad\hat{A}_{k}^{(i)}=r_{k}^{(i)}. This is an unbiased estimator for state-action function which typically has high variance. 2. Batch Mean Baseline. Using all samples in the current batch (across prompts), \hat{b}_{k}^{(i)}=\mathbb{E}_{j\sim\text{batch}}[\,r_{k}^{(j)}\,]. This reduces variance, but may be poorly matched to a given prompt’s difficulty and to the specific prefix X_{1 because rewards are averaged across trajectories with different intermediate prefixes. This is the original proposition from GRPO work how it can be applied in multi-turn scenarios. 4. Outcome-Conditioned Baseline (OCB). Partitioning the group into subgroups based on a discrete intermediate outcome o^{(i)} of the prefix X_{ tags. The final answer is enclosed between tags. The analysis about confidence and uncertainty is enclosed within tags. The Assistant should reason about its confidence in the solution and its uncertainty in the solution within these tags. The final format that must be followed is: reasoning process here final answer here analysis about confidence and uncertainty here confidence level here (number between 0 and 1) EXAMPLES Q: Who was awarded the Oceanography Society’s Jerlov Award in 2018? The question asks who was awarded the Oceanography Society’s Jerlov Award in 2018. The Jerlov Award is given by the Oceanography Society, and it is named after the late Professor Gunnar Jerlov, a pioneer in the field of ocean optics. To find the winner of the 2018 award, I would need to look up the most recent information on the Oceanography Society’s website or other reliable sources. Dr. David W. Schiedek The Oceanography Society’s website or other reliable sources do not provide information on the winner of the 2018 Jerlov Award. It is possible that the information is not readily available or that the award was not given in 2018. Additionally, the name ”Dr. David W. Schiedek” is not a well-known figure in the field of oceanography, which raises uncertainty about the accuracy of this answer. 0.25 END OF EXAMPLES The Two-Attempt Refinement experiment was run with the following system prompt. Two-Attempt Refinement A conversation between User and Assistant. The user asks a question, and the Assistant solves it using multiple attempts. For each attempt, the Assistant thinks about the reasoning process and provides an answer. Each attempt consists of reasoning enclosed in tags followed by an answer enclosed in tags. After the first attempt, the Assistant reflects on the previous solution and may provide an improved answer. The format for 2 attempts is: first attempt reasoning first attempt answer reflection and second attempt reasoning second attempt answer (same or corrected) EXAMPLES Q: What is 2 + 2? This is a simple arithmetic problem. 2 + 2 equals 4. 4 Let me verify: 2 + 2 = 4. This is correct. 4 Q: Who was awarded the Oceanography Society’s Jerlov Award in 2018? The question asks who was awarded the Oceanography Society’s Jerlov Award in 2018. The Jerlov Award is given by the Oceanography Society, and it is named after the late Professor Gunnar Jerlov, a pioneer in the field of ocean optics. To find the winner of the 2018 award, I would need to look up the most recent information on the Oceanography Society’s website or other reliable sources. Dr. David W. Schiedek Let me reconsider. The Oceanography Society’s website or other reliable sources do not provide information on the winner of the 2018 Jerlov Award under the name Dr. David W. Schiedek. However, there is information that the award was given to Dr. Annick Bricaud in 2018, which sounds more plausible given her contributions to ocean optics. Dr. Annick Bricaud END OF EXAMPLES Table 4 : Canonical configuration shared by all runs (YAML-like rendering for readability). # Common settings (all GRPO-like runs) sequence_length: 4096 temperature: 1.0 batch_size: 2048 group_size: 32 # unless stated otherwise optimizer: AdamW adam_beta1: 0.9 adam_beta2: 0.999 weight_decay: 0.1 lr: 1e-6 warmup: linear(64 steps) # Dynamic entropy controller target_entropy: 0.2 eta_delta: 5e-4 kl_beta: 0.0 # we don’t use KL regularization weights_dtype: bfloat16 Appendix B Additional Method Details Algorithm 1 BAE with outcome-conditioned baselines (for one prompt x ) 1: Sample a GRPO group \{y^{(i)}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x) and parse each completion into blocks y^{(i)}=[X_{1}^{(i)};\ldots;X_{K}^{(i)}] . 2: Compute per-block rewards \{r_{k}^{(i)}\} and intermediate outcomes o_{k}^{(i)}=o_{k}(x,X_{1 . 3: for k=1 to K do 4: if k=1 then 5: Set baseline \hat{b}_{1}^{(i)}\leftarrow\frac{1}{G}\sum_{j=1}^{G}r_{1}^{(j)} . 6: else 7: For each outcome value o , compute stratum mean \mu_{k,o}\leftarrow\frac{1}{|\mathcal{G}_{o}|}\sum_{j\in\mathcal{G}_{o}}r_{k}^{(j)} where \mathcal{G}_{o}=\{j:o_{k}^{(j)}=o\} . 8: Set \hat{b}_{k}^{(i)}\leftarrow\mu_{k,o_{k}^{(i)}} . 9: end if 10: Form proxy-advantage \hat{A}_{k}^{(i)}\leftarrow r_{k}^{(i)}-\hat{b}_{k}^{(i)} . 11: end for 12: Update \theta by optimizing the blockwise clipped objective, applying \hat{A}_{k}^{(i)} only to tokens in block k . When is OCB unbiased? For a fixed prompt x and block k , suppose there exists a discrete outcome variable o=o(x,X_{ To find the length of segment BN, I can use the properties of similar triangles. Since MN is parallel to AB, triangles CAN and CBM are similar, where side AC corresponds to side BC and side AN corresponds to side BM. Using the property of similar triangles, we have the proportion: AC/BC = AN/BM We know that AC is 5 units long, BC is 2.4 units long, and AN is 3 units long. We can find the length of BM by using the fact that AC + AN = AP (where AP is the sum of the lengths of segments AC and AN). So, BM = AP - AM - MC. Knowing these values, I can substitute them into the proportion and solve for the length of BM. 11.5 Given that MN is parallel to AB, triangles CAN and CBM are similar, and the proportions can be set up with the given lengths. Substituting the values and solving for BM gives the required length of BN. However, since I used approximate values for AC and BM, there may be some uncertainty in the final answer, and thus the confidence in the solution is relatively low. 0.6 Appendix F GRPO Calibration Trained with BCE Reward Here we illustrate that a naive reward scheme for standard GRPO, R=r_{\mathrm{RLVR}}+r_{\mathrm{BCE}}, can lead to a local optimum where the model learns to refuse to answer while minimizing its confidence. This behavior can be interpreted as a form of reward hacking. Mitigating it would require hyperparameter tuning or the introduction of additional terms into the reward. RLVR + BCE Reward Let a and b be positive integers satisfying 3a I am not familiar with the equation presented in the question, involving positive integers a and b. I know what positive integers are, but I do not recognize this specific equation or how to solve it. Additionally, I am unsure about the context of the question, as it involves a symbol ( \exists ) and a square that looks like an algebraic expression, but it is not clear to me what it means. Unable to determine This question involves a complex equation with a symbol and a square that I am unfamiliar with. There are no guidelines or previous examples to follow, and the equation does not resemble any standard mathematical format that I am aware of. The lack of context and familiarity with the equation make it impossible to provide an accurate answer or even a plausible guess. 0.001 In contrast, OCB with BCE optimizing the confidence block shows no signs of reward hacking. OCB with BCE Reward For Confidence Block In the circle with center Q , radii AQ and BQ form a right angle. The two smaller regions are tangent semicircles, as shown. The radius of the circle with center Q is 14 inches. What is the radius of the smaller semicircle? Express your answer as a common fraction. To find the radius of the smaller semicircles in the given geometric configuration, we need to follow these steps: 1. Understand the Initial Setup : - There’s a circle centered at Q with radius 14 inches. - Radii AQ and BQ form a right angle. - Two smaller regions are tangent semicircles, as shown. 2. Visualize the Problem : - The circle with center Q provides a structural guide for the configuration. - Radii AQ and BQ are perpendicular to each other and hence form a right angle at Q . - The two semicircles are tangent to each other and tangential to the larger circle’s boundary. 3. Establish Relationships and Names : - Let the radius of the larger circle be R=14 inches. - Let the radius of each of the smaller semicircles be r . 4. Analyze the Geometry : - The right angle formed by AQ and BQ divides the larger circle into four sectors. - Each smaller semicircle is tangent to the larger circle and each other at points along the radius of the larger circle. 5. Use Geometric Properties : - The combined diameter of the two smaller semicircles is equal to the diameter of the larger circle minus the diameter of each semicircle. - The part of the radius of Q occupied by the semicircles traces the perimeter of the square inscribed within the circle. 6. Set Up the Equations : - The diagonal of the inscribed square is the diameter of the circle. - The diameter of the circle is 2R=2\times 14=28 inches. - For an inscribed square within the circle, the length of one side of the square can be denoted as a . The diagonal of the square forms a right triangle with two sides of length a each: a\sqrt{2}=2R\implies a\sqrt{2}=28\implies a=\frac{28}{\sqrt{2}}=14\sqrt{2}. - The area of the square is a^{2}=(14\sqrt{2})^{2}=392 square inches. 7. Apply Tangency Conditions : - The tangency condition requires the semicircles to align appropriately within the larger circle. - Considering the sum of the diameters of the two semicircles along with the right angle setup mentioned, the combined diameter of the two smaller semicircles equals the remaining radius of the larger circle after accounting for the diameter of the square portion. 8. Calculate the Radius of the Smaller Semicircles : - Each semicircle with radius r has a diameter of 2r . - The combined diameter of the two semicircles is 2r+2r=4r . - This combined diameter needs to fit into the space remaining within the circle as its larger circle portion: 4r=14-\text{(One side of square per radius)}=14+14-14=14\sqrt{2}-14\implies 4r=14. - Solving for r : r=\frac{14}{4}=\frac{7}{2}. Therefore, the radius of the smaller semicircle is \frac{7}{2} inches. \frac{7}{2} First, let’s verify and refine the geometric configuration and relationships we have worked with. 1. Circle and Radii Setup : We have a circle centered at Q with radius R=14 inches. Radii AQ and BQ are perpendicular, splitting the circle into four quadrants. 2. Semicircles Positioning : The smaller semicircles are tangent to each other and tangent to the larger circle’s boundary. However, the radius ratio and placements need clear geometric consideration. 3. Geometry Revisit : Let the radius of the smaller semicircles be r . The two tangent semicircles form a width equal to their combined diameters along AQ and BQ . The radius R of the larger circle is composed of segments r+r along with another segment equal to r in one direction. 4. Right Triangle Insight : Given that the two semicircles form a right angle at Q , and the semicircles’ radius sum along with the square inscribed within provides the key geometric property: - The diameters 2r of each semicircle sum up to 2r+2r=4r . - The diagonal of the square inscribed in the circle of radius 14 inches is 2R=28 inches, forming the right triangle with a\sqrt{2}=28 , giving a=14\sqrt{2} . The area calculation as a^{2}=392 square inches in this context adds insignificant value for finding the smaller semicircles’ radii. 5. Mistakes and Clarifications : The linking of squared terms and square side lengths probably needs better connection. The right triangle and square properties were considered but not correctly tied to the radius r . Taking a simpler approach, realizing the relationship directly between the inscribed squares and semicircles, let’s re-center: - The diameters of the semicircles and their tangency precisely dictate the relationship with the large circle. - Confirmation through an initial insight that 2r equates to R , r=\dfrac{R}{3} . Finally, upon thorough review and correction of the geometric relationships and calculations: r=\frac{14}{3}. 0.15 Appendix G Accuracy/Calibration Contribution into TTS Performance When calibration is “good enough”, accuracy dominates TTS. A consistent empirical pattern is that once a model is already reasonably calibrated (e.g., ECE on the order of a few percent), marginal improvements in calibration metrics do not necessarily translate into better test-time scaling unless they also improve (or at least preserve) solution accuracy and sample quality. For example, in our 7B-Base experiments (Table 2 ), the group-mean baseline achieves lower ECE than OCB on MATH500, yet it yields worse TTS accuracy under confidence-weighted ensembling because its Pass@1 accuracy is substantially lower. This suggests that in the “well-calibrated” regime, the dominant driver of TTS is the quality/diversity of sampled solutions (accuracy and Pass@k headroom), while confidence primarily serves as a tie-breaker and robustness mechanism. Appendix H Group Size Ablation for OCB Method Outcome-Conditioned Baselines estimate the conditional value for later blocks by averaging rewards within outcome-defined subgroups of a GRPO group (e.g., correct vs. incorrect solutions). This estimator therefore depends on the group size G : larger groups typically provide more samples per outcome class, reducing variance of the subgroup mean but increasing sampling cost. We study the effect of group size on OCB using Qwen2.5-7B-Base trained for 512 steps with the same batch size of 2048 completions. All settings are identical except for the GRPO group size, comparing G=32 and G=64 . Table 5 shows that increasing the group size from 32 to 64 yields slightly improved calibration (lower ECE) on all three benchmarks, while causing a reduction in accuracy and AUROC in our runs. We attribute this to a twofold lower number of distinct examples seen during training. Table 5 : Effect of GRPO group size G for BAE with Outcome-Conditioned Baselines (OCB) on Qwen2.5-7B-Base (512 steps). Larger groups slightly reduce ECE but can trade off against accuracy and discrimination (AUROC). Setting Acc (%) AUROC ECE Brier MATH500 (in-distribution) OCB ( G{=}32 ) 75.1 0.92 0.03 0.09 OCB ( G{=}64 ) 75.1 0.90 0.03 0.09 GSM8K (easy OOD) OCB ( G{=}32 ) 89.7 0.85 0.04 0.07 OCB ( G{=}64 ) 88.4 0.81 0.02 0.08 AIME23–25 (hard OOD) OCB ( G{=}32 ) 9.5 0.90 0.13 0.06 OCB ( G{=}64 ) 8.6 0.89 0.12 0.07 Table 6 : Main results for Qwen2.5-7B-Instruct trained for 512 steps on self-confidence estimation for math. Method Acc (%) AUROC ECE Brier MATH500 (in-distribution) RLCR 78.6 0.922 0.031 0.086 OCB 77.3 0.929 0.040 0.087 Group Mean 76.6 0.893 0.032 0.098 GSM8K (easy OOD) RLCR 89.8 0.760 0.032 0.081 OCB 89.1 0.827 0.032 0.076 Group Mean 82.4 0.641 0.125 0.146 AIME23–25 (hard OOD) RLCR 11.9 0.932 0.132 0.072 OCB 12.9 0.907 0.096 0.074 Group Mean 12.1 0.937 0.094 0.058 Appendix I Full Results with Standard Errors Table 7 reports evaluation metrics in mean \pm SEM format. Uncertainty estimation for evaluation metrics. We report standard errors of the mean (SEM) for all evaluation metrics to quantify estimation uncertainty. For Pass@1, we compute SEM directly from the empirical distribution of metric values across sampled combinations. the SEM is computed as \sigma/\sqrt{n} , where \sigma is the sample standard deviation and n is the total number of combination-level evaluations. For calibration metrics (Expected Calibration Error, Brier score, and AUROC), which are computed globally across all predictions, we employ bootstrap resampling with 1,000 iterations. In each bootstrap iteration, we resample the (q,c) pairs (confidence, correctness) with replacement and recompute each calibration metric; the SEM is estimated as the standard deviation of the resulting bootstrap distribution. Table 7 : Evaluation metrics with standard errors (mean \pm SEM). Model Dataset Method Acc (%) AUROC ECE Brier Qwen2.5-3B Instruct MATH500 (in-dist.) RLCR 68.65 \pm 1.04 0.8814 \pm 0.0085 0.0591 \pm 0.0071 0.1147 \pm 0.0046 Group Mean 66.35 \pm 1.06 0.8781 \pm 0.0086 0.0688 \pm 0.0074 0.1241 \pm 0.0056 OCB 67.90 \pm 1.04 0.8927 \pm 0.0081 0.0297 \pm 0.0064 0.1104 \pm 0.0046 Batch Mean 66.15 \pm 1.06 0.8584 \pm 0.0090 0.0775 \pm 0.0074 0.1283 \pm 0.0044 None (raw reward) 66.10 \pm 1.06 0.8818 \pm 0.0083 0.0990 \pm 0.0076 0.1248 \pm 0.0061 Qwen2.5-7B Base MATH500 (in-dist.) RLCR 74.99 \pm 0.48 0.9140 \pm 0.0037 0.0426 \pm 0.0033 0.0932 \pm 0.0021 OCB 75.10 \pm 0.48 0.9169 \pm 0.0035 0.0322 \pm 0.0030 0.0935 \pm 0.0020 Group Mean 72.71 \pm 0.50 0.9117 \pm 0.0037 0.0199 \pm 0.0032 0.0991 \pm 0.0021 GSM8K (easy OOD) RLCR 88.86 \pm 0.22 0.8345 \pm 0.0047 0.0368 \pm 0.0018 0.0753 \pm 0.0013 OCB 89.70 \pm 0.21 0.8473 \pm 0.0046 0.0394 \pm 0.0019 0.0729 \pm 0.0012 Group Mean 87.77 \pm 0.23 0.8487 \pm 0.0044 0.0297 \pm 0.0019 0.0793 \pm 0.0013 AIME23–25 (hard OOD) RLCR 9.41 \pm 0.77 0.9416 \pm 0.0115 0.0920 \pm 0.0053 0.0518 \pm 0.0031 OCB 9.48 \pm 0.78 0.9018 \pm 0.0180 0.1263 \pm 0.0055 0.0639 \pm 0.0035 Group Mean 8.22 \pm 0.73 0.9105 \pm 0.0141 0.0941 \pm 0.0055 0.0563 \pm 0.0033 Qwen2.5-7B Instruct MATH500 (in-dist.) RLCR 78.55 \pm 0.46 0.9215 \pm 0.0036 0.0305 \pm 0.0030 0.0864 \pm 0.0022 OCB 77.30 \pm 0.47 0.9293 \pm 0.0031 0.0395 \pm 0.0031 0.0867 \pm 0.0023 Group Mean 76.59 \pm 0.47 0.8926 \pm 0.0043 0.0323 \pm 0.0031 0.0979 \pm 0.0024 GSM8K (easy OOD) RLCR 89.78 \pm 0.21 0.7595 \pm 0.0055 0.0318 \pm 0.0019 0.0813 \pm 0.0016 OCB 89.11 \pm 0.21 0.8269 \pm 0.0050 0.0315 \pm 0.0019 0.0759 \pm 0.0016 Group Mean 82.43 \pm 0.52 0.6413 \pm 0.0084 0.1247 \pm 0.0047 0.1459 \pm 0.0043 AIME23–25 (hard OOD) RLCR 11.94 \pm 0.86 0.9320 \pm 0.0103 0.1319 \pm 0.0061 0.0718 \pm 0.0035 OCB 12.85 \pm 0.89 0.9073 \pm 0.0124 0.0957 \pm 0.0065 0.0739 \pm 0.0042 Group Mean 12.08 \pm 1.73 0.9365 \pm 0.0219 0.0936 \pm 0.0110 0.0581 \pm 0.0067