Real-Time Aligned Reward Model beyond Semantics

Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuefeng Xiao, Hongyan Xie, Li Huaqiu, Songshi Liang, Zhongxiang Dai, Fuzhen Zhuang, Jianxin Li, Yikun Ban, Deqing Wang · 2026

Read Real-Time Aligned Reward Model beyond Semantics by Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuefeng Xiao, Hongyan Xie, Li Huaqiu, Songshi Liang, Zhongxiang Dai, Fuzhen Zhuang, Jianxin Li, Yikun Ban, Deqing Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22664.txt.

Introduction

Zixuan Huang and a team align reward models with human taste in real time, past mere word sense. RLHF trains models from human likes, yet the policy can overfit the reward model and game its quirks. Their 2026 preprint targets that reward overoptimization, where scores rise while true quality stalls.

Their reward model updates as it is used, staying in step with the policy it grades. A live, aligned grader should cut gaming and drift. The work matters for every lab that trains by feedback. It treats alignment as a moving target. That honesty is the point.

Catalog record

CREATORZixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuefeng Xiao, Hongyan Xie, Li Huaqiu, Songshi Liang, Zhongxiang Dai, Fuzhen Zhuang, Jianxin Li, Yikun Ban, Deqing Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22664

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-Time Aligned Reward Model beyond Semantics Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuefeng Xiao, Hongyan Xie, Li Huaqiu, Songshi Liang, Zhongxiang Dai, Fuzhen Zhuang, Jianxin Li, Yikun Ban, Deqing Wang Abstract Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily rely on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M ( R eal-Time Aligned R eward M odel), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback ) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models. 1 Introduction Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone technique for aligning large language models (LLMs) with human values and preferences ( 50 ; 26 ; 46 ; 45 ; 24 ) . However, RLHF faces a persistent challenge: reward overoptimization. Instead of faithfully capturing human intent, policy models often exploit spurious reward patterns, such as response length, markdown formatting, or superficial linguistic cues like certain n-grams or emojis, to maximize rewards without genuinely improving alignment ( 20 ; 14 ; 19 ) . The core issue lies in the reward model: trained on limited preference data, it can only approximate human values. As the policy evolves during RLHF training while the reward model remains fixed, distribution shift exacerbates approximation errors ( 52 ) , ultimately leading to unreliable reward signals in optimization. A common mitigation is to iteratively update the reward model so that it adapts to the policy’s evolving behavior. Yet, direct retraining of the reward model at each iteration is computationally prohibitive. To address this, one research direction emphasizes uncertainty-aware corrections. 14 ; 19 ; 61 penalize uncertain samples during policy training, while 63 introduce kernel-based uncertainty estimates derived from reward model embeddings. Another line of work focuses on robust reward model retraining. 30 incorporate an unsupervised mutual information loss to counter distribution shift, and 34 augment training data by decomposing preferences relative to prompts. These methods trade off efficiency and robustness, but leave open a critical question: Can we design a new RLHF framework that preserves training efficiency while effectively achieving real-time alignment of reward models towards policy models? We propose R2M ( R eal-Time Aligned R eward M odel) to solve this challenge. It is a lightweight RLHF framework in which the RM itself is reinforced iteratively by dynamically adapting to the policy’s internal states, and it does not require any additional labeled data or environmental feedback to improve the performance. Specifically, we observe that deep-layer hidden states of the policy encode latent patterns that are closely correlated with both golden human preferences and the scalar reward scores assigned by RMs. This observation aligns with the perspectives on implicit reward modeling advanced in works such as DPO ( 39 ) and PRIME ( 16 ) , yet it is often overlooked by existing explicit reward models ( 34 ; 63 ) . Building on this insight, we aim to go beyond reward models that solely depend on semantic representations of a pretrained LLM. Instead, we enhance the reward model by incorporating the evolving hidden states of the policy model (namely policy feedback ). To this end, we redesign the scoring head of the RM so that it dynamically integrates these hidden states, enabling the RM to adapt to distribution shifts in the policy. In our RLHF framework, this introduce a lightweight training component that learns to aggregate policy feedback directly, enhancing the RM’s representation without retraining the entire model. Owing to its efficiency, this mechanism can be seamlessly applied at every training round, ensuring continuous synchronization between the reward model and the policy model. The design of R2M offers two benefits: 1) Iterative distribution alignment with accurate reward allocation. The reward model integrates the policy’s evolving hidden states which provide behaviorally grounded and semantically informed feedback. This mitigates distribution shifts, reduces reward overoptimization, and ensures more accurate reward assignment. 2) Extremely lightweight overhead. R2M only need to learn how to aggregate representations, introducing negligible additional cost. Experimental results demonstrate that R2M significantly improves performance on dialogue tasks (trained on UltraFeedback ( 15 ) , evaluated on Alpaca-Eval ( 18 ) ) and text summarization tasks (trained and evaluated on TL;DR summarization dataset). Specifically, compared with vanilla RLOO, RLOO+R2M increases the AlpacaEval 2 win rate (WR) by 5.2% - 8.0%, the length-controlled win rate (LC) by 2.9% - 6.1% and the TL;DR win rate by 6.3% compared to baselines, while introducing only minimal computational cost. Furthermore, we conducted a comprehensive analysis, showing that R2M effectively strengthens the vanilla RM and mitigates reward overoptimization with minimal additional training overhead. 2 Preliminary RLHF consists of three main steps: 1) Supervised Fine Tuning, 2) Reward Modeling, and 3) RL optimization. We provide a detailed workflow shown in Appendix H.1 . As R2M is designed to directly integrated into the RL optimization phase, let us consider the following typical third-stage RL Optimization process: Trajectory Sampling : At each training step t\in[T] , we update offline policy \pi_{old} to online policy \pi_{\theta} . Then, given a query set X_{t}=\{x_{1},x_{2},\dots,x_{n}\} , \pi_{old} is used to sample a group of K responses G_{i}=\{y_{i,j}\}_{j=1}^{K} for each x_{i}\in X_{t} . Reward Annotation : For each (x_{i},G_{i}),i\in[n] , there are K query-response pairs (x_{i},y_{i,j}),j\in[K] . We use a scalar reward model r_{\varphi}(x,y) to assign scores to each query-response pair, obtaining \{r_{i,j}|i\in[n],j\in[K]\} , resulting in a batch \mathcal{B}=\{(x_{i},y_{i,j},r_{i,j})|i\in[n],j\in[K]\} . After this process, we employ the RLOO approach ( 2 ) to perform advantage estimation within each G_{i} : \hat{A}_{i,j}=r_{i,j}-\frac{1}{K-1}\sum_{\hat{j}\neq j}r_{i,\hat{j}}. (1) Policy Optimization : For each query-response pair (x_{i},y_{i,j}) , we perform a forward pass in the policy model \pi_{\theta} and optimize \pi_{\theta} using importance sampling by maximizing the following objective ( 44 ; 2 ) , where \varepsilon and \beta are hyperparameters: \displaystyle\min[\frac{\pi_{\theta}(y_{i,j}|x_{i})}{\pi_{\theta_{\text{old}}}(y_{i,j}|x_{i})}\hat{A}_{i,j}, (2) \displaystyle\text{clip}(\frac{\pi_{\theta}(y_{i,j}|x_{i})}{\pi_{\theta_{\text{old}}}(y_{i,j}|x_{i})},1-\varepsilon,1+\varepsilon)\hat{A}_{i,j}\left.\right] \displaystyle-\beta\mathbb{D}_{KL}[\pi_{\theta}\|\pi_{\text{ref}}] The design of R2M is based on the aforementioned RL optimization process. As a lightweight and significantly effective alternative, R2M can be seamlessly deployed to all REINFORCE-based RLHF frameworks. Due to resource constraints, we adopt RLOO as one of the primary baselines. 3 Motivation We argue that deep-layer hidden states of the policy in a transformer’s forward pass contain crucial information that are closely correlated with both golden human preferences and reward scores, making them effective for enhancing vanilla RMs. Due to space constraints, the experimental details of this section are provided in the Appendix G.1 . Figure 1 establishes the relationship between hidden state similarity and preference labels. The average hidden state similarity between pairs with different preference labels

Read the complete text