Alternating Reinforcement Learning with Contextual Rubric Rewards
Guangchen Lan · 2026
Read Alternating Reinforcement Learning with Contextual Rubric Rewards by Guangchen Lan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15646.txt.
Introduction
Lan and co-authors rework how rubric rewards guide reinforcement learning. Rubric methods score answers on several traits at once. Fixed weighting then crushes those scores into one number. That step is touchy and hides links between traits. Their ARL-RR method trains on one rubric class at a time. A light search picks the next class from task results. On HealthBench, the method beat scalarized rivals at four model sizes. It also trained faster in their tests.
Catalog record
| CREATOR | Guangchen Lan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56876 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Alternating Reinforcement Learning with Contextual Rubric Rewards Guangchen Lan Abstract Reinforcement Learning with Rubric Rewards (RLRR) is a framework that extends conventional reinforcement learning from human feedback (RLHF) and verifiable rewards (RLVR) by replacing scalar preference signals with structured, multi-dimensional, contextual rubric-based evaluations. However, existing approaches in RLRR are limited to linearly compressing vector rewards into a scalar reward with a fixed weightings, which is sensitive to artificial score design and fails to capture correlations among reward dimensions. To overcome the limitations of reward aggregation, this work proposes Alternating Reinforcement Learning with Rubric Rewards (ARL-RR), a framework that eliminates the need for a fixed scalarization by optimizing one semantic rubric meta-class at a time. Theoretically, we show that reward aggregation induces a variance contraction effect, which helps explain the performance gains. We further introduce a lightweight, search-based adaptation procedure that selects the next meta-class dynamically based on task performance, enabling the policy to emphasize critical objectives and thereby improve the model performance. Empirically, our experiments on the HealthBench dataset with experts annotations demonstrate that ARL-RR uniformly outperforms scalarized methods in both model performance and training efficiency across different model scales (1.7B, 4B, 8B, and 14B). 1 Introduction Reinforcement Learning (RL) has emerged as a central paradigm for aligning large language models (LLMs) with human preferences ( 23 ) and improving reasoning abilities ( 9 ) . Pioneering work framed the problem as reinforcement learning (RL) on a reward model trained from a batch of comparisons, yielding notable improvements in games ( 6 ) , summarization ( 29 ) , mathematics ( 27 ) , and safety ( 13 ) . While conventional RL from Human Feedback (RLHF) and RL with Verifiable Rewards (RLVR) typically rely on scalar preference signals, recent advancements have moved toward richer and more structured evaluations. Reinforcement Learning with Rubric Rewards (RLRR) extends these frameworks by replacing simple scalar signals with multi-dimensional, rubric-based assessments ( 8 ; 12 ) . Rather than collapsing quality into a single scalar, RLRR structures evaluation around explicit criteria, such as accuracy, completeness, and instruction following ( 11 ) , yielding more informative feedback for credit assignment and diagnosis. This criterion-level structure provides a more granular view of model behavior that rule-based rewards alone may not fully capture ( 22 ) . However, a critical limitation exists in current methodologies: existing approaches generally linearly compress these vector rewards into a single scalar reward using fixed weightings. Recent studies on Directional Preference Alignment (DPA) ( 33 ) and Pareto Multi-Objective Alignment (PAMA) ( 10 ) have highlighted that this “scalarization strategy” is highly sensitive to artificial score design and fails to capture the complex, non-linear correlations among different reward metrics, potentially leading to reward hacking ( 21 ) . By aggregating distinct signals, the variance and entropy of the feedback are often dampened, causing the policy to lose the nuance required for multi-objective optimization. Consequently, the policy is forced to optimize a crude approximation of the desired behavior, potentially sacrificing performance on difficult metrics, e.g., accuracy, to maximize easier ones, e.g., completeness ( 34 ) . Consequently, this raises a pivotal question: How can we optimize LLMs with contextual rubric rewards without a fixed linear scalarization, in a way that preserves the structure and diversity of per-criterion feedback? We answer the above question by proposing Alternating Reinforcement Learning with Rubric Rewards (ARL-RR), a framework for policy optimization that eliminates the need for scalarization and overcomes the limitations of linear reward aggregation in RLRR. Instead of compressing objectives, ARL partitions criteria into several meta-classes, including accuracy, completeness, instruction following, context awareness, and communication quality, and alternates optimization across these rubric meta-classes. This approach leverages the insight that diverse rollout responses yield better RL performance ( 1 ; 18 ; 4 ) by preserving the inherent diversity of feedback signals within specific semantic dimensions. Furthermore, we introduce contextual rubric adaptation via a search-based strategy that dynamically selects the active rubric meta-class based on task performance. This enables the policy to emphasize critical objectives during training, effectively transforming the multi-objective problem into a sequence of focused sub-problems. Empirically, our experiments on the HealthBench ( 2 ) dataset demonstrate that ARL-RR uniformly outperforms scalarized methods in both model performance and training efficiency across different model sizes, ranging from Qwen3-1.7B to 14B. Together, these contributions advance the efficiency and effectiveness of LLM post-training beyond conventional scalarized aggregation schemes. 1.1 Our Contributions In summary, the main contributions of this work are: 1. We introduce a novel RLRR training method for LLMs. We propose ARL-RR, a framework that bypasses fixed linear weighting schemes to effectively capture the structure of multi-dimensional evaluation criteria. 2. Meta-class pipeline. We design a comprehensive pipeline that maps contextual rubric criteria into a fixed set of meta-classes (accuracy, completeness, instruction following, context awareness, and communication quality), enabling consistent training in contextual multi-objectives. 3. Meta-class searching. We develop a lightweight search-based strategy for contextual rubric adaptation, allowing the model to dynamically prioritize specific meta-classes (e.g., accuracy or completeness) to maximize performance. 4. Theoretical analysis. We show that linear scalarization contracts reward variance across rollout samples under mild correlation assumptions, explaining why scalarized rewards can weaken Monte Carlo based (rollout) learning signals. 5. Effectiveness. We demonstrate that ARL-RR consistently outperforms scalarized RL baselines on the HealthBench dataset across various model scales (Qwen3-1.7B to 14B), achieving higher evaluation scores. 6. Efficiency. Our method improves training efficiency, achieving better performance in fewer epochs compared to scalarized approaches. 2 Background Reinforcement Fine-Tuning. First, we introduce the general framework of RL. Consider the Markov decision process (MDP) as a tuple (\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}) , where \mathcal{S} is the state space, \mathcal{A} is a finite action space, \mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1] is a Markov kernel that determines transition probabilities, and \mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is a reward function. At each time step t , the agent executes an action a_{t}\in\mathcal{A} from the current state s_{t}\in\mathcal{S} , following a stochastic policy \pi , i.e. , a_{t}\sim\pi(\cdot|s_{t}) . The corresponding reward is defined as r_{t} . Following the conventional setting in LLMs, the policy \pi_{\theta} represents the LLM with model parameters \theta . The action space \mathcal{A} is set as the vocabulary. At step t , s_{t}=(q,a_{<t}) is a cascade of the query q and the tokens a_{<t}=(a_{1},\cdots,a_{t-1}) that have been predicted, and a_{t} is the next token to be predicted. The transition kernel \mathcal{P} is deterministic as s_{t+1}=(s_{t},a_{t}) . The complete answer a=(a_{1},\cdots,a_{T}) with length |a|=T . The step reward r_{t}=R(q,a_{\leq t}) can be obtained from a rule-based reward function ( 22 ) or a trained reward model. To optimize the policy \pi_{\theta} , in GRPO ( 27 ) , it is suggested to optimize the model \theta by maximizing the objective function in each update as follows: \displaystyle J(\theta)=\mathop{\mathbb{E}}_{q\sim\mathcal{D},\atop\{a^{i}\}_{i=1}^{G}\sim\pi_{\rm old}(\cdot|q)}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|a^{i}|}\sum_{t=1}^{|a^{i}|}\Big(\min\big(\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\rm old}(a_{t}|s_{t})}A_{i},{\rm clip}(\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\rm old}(a_{t}|s_{t})},1-\epsilon,1+\epsilon)A_{i}\big)-\beta D_{\rm KL}(\pi_{\theta}\|\pi_{\rm ref})\Big)\Big], (1) where \pi_{\rm ref} is the reference policy with the initial model parameters, \pi_{\rm old} is the old policy with the parameters before this update, \mathcal{D} is the prompt data set, G is the group (rollout) size, \beta is a hyperparameter to control the weight of the Kullback–Leibler (KL) divergence, \epsilon is a hyperparameter to control the clip ratio, and {\rm clip}(\cdot) is a clip function following the setting in PPO ( 25 ) .