Factored Causal Representation Learning for Robust Reward Modeling in RLHF

Yupei Yang, Lin Yang, Wanxi Deng, Lin Qu, Fan Feng, Biwei Huang, Shikui Tu, Lei Xu · 2026

Read Factored Causal Representation Learning for Robust Reward Modeling in RLHF by Yupei Yang, Lin Yang, Wanxi Deng, Lin Qu, Fan Feng, Biwei Huang, Shikui Tu, Lei Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21350.txt.

Introduction

Yupei Yang and seven co-authors fix a weakness in the reward models that steer language models toward human tastes. A reward model scores answers so the main model can learn from feedback. Standard reward models can latch onto surface traits that are not truly linked to what people liked.

That flaw invites reward hacking, where high scores stop meaning better behavior. The authors split the model's inner features into causal parts and spurious parts. Training leans on the causal parts, so the score tracks real quality. A steadier reward signal should make feedback training safer and more honest.

Catalog record

CREATORYupei Yang, Lin Yang, Wanxi Deng, Lin Qu, Fan Feng, Biwei Huang, Shikui Tu, Lei Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21350

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Factored Causal Representation Learning for Robust Reward Modeling in RLHF Yupei Yang, Lin Yang, Wanxi Deng, Lin Qu, Fan Feng, Biwei Huang, Shikui Tu, Lei Xu Abstract A reliable reward model is essential for aligning large language models (LLMs) with human preferences through reinforcement learning from human feedback (RLHF). However, standard reward models are susceptible to spurious features that are not causally related to human labels. This can lead to reward hacking , where high predicted reward does not translate into better behavior. In this work, we address this problem from a causal perspective by proposing a factored representation learning framework that decomposes the model’s contextual embedding into (1) causal factors that are sufficient for reward prediction and (2) non-causal factors that capture reward-irrelevant attributes such as length or sycophantic bias. The reward head is then constrained to depend only on the causal component. In addition, we introduce an adversarial head trained to predict reward from the non-causal factors, while applying gradient reversal to discourage them from encoding reward-relevant information. Experiments on both mathematical and dialogue tasks demonstrate that our method learns more robust reward models and consistently improves downstream RLHF performance over state-of-the-art baselines. Analyses on length and sycophantic bias further validate the effectiveness of our method in mitigating reward hacking behaviors. † † affiliationnotice: \dagger This work was done when the author was a research intern at Alibaba Group. 1 Introduction In recent years, RLHF has emerged as a powerful approach for aligning LLMs with human preferences ( 36 ; 4 ) . A core component of RLHF is the reward model, which serves as a proxy for human judgment during policy optimization. However, standard reward model training is prone to learning spurious correlations: the model may assign higher scores to preference-irrelevant patterns, such as response length or sycophantic phrasing, rather than the true causal drivers of human preference ( 28 ; 32 ; 56 ) . The LLM then exploits these shortcuts during RLHF, achieving higher predicted rewards while drifting from the intended objective—a phenomenon known as reward hacking ( 2 ; 17 ; 49 ) . To tackle this issue, extensive efforts have been made to mitigate the influence of known spurious factors in reward modeling. For example, ODIN ( 8 ) decomposes the reward head into separate quality and length components to reduce reward hacking driven by response length. 38 develops a length-regularization strategy to prevent length exploitation during DPO ( 44 ) . 56 further proposes a more general maximum mean discrepancy (MMD)-based regularization that can constrain spurious behaviors beyond length, such as sycophantic bias ( 47 ) or concept bias ( 65 ) . Despite their effectiveness, these approaches require explicitly specifying the spurious variables to be controlled, whereas it is challenging to anticipate all possible exploitation patterns in practice. Another line of work seeks to improve reward models by filtering out irrelevant information directly through representation learning. 35 employs a contrastive objective to learn goal-conditioned representations within the reward model, which helps distinguish between preferred and dispreferred responses. InfoRM ( 32 ; 31 ) adopts an information-theoretic perspective, introducing a variational information bottleneck objective to encourage the latent representations to retain only information relevant to human preference. However, to the best of our knowledge, none of the existing methods explicitly disentangle reward-irrelevant factors from the latent state for reward modeling. Nevertheless, causal representation learning ( 45 ; 23 ; 26 ; 59 ) has demonstrated strong promise in traditional reinforcement learning for learning minimal sufficient state representations that capture only the causally relevant aspects of the environment. Motivated by these insights, we propose CausalRM , a novel framework that explores the potential of causal representation learning to mitigate spurious correlations in reward modeling. Specifically, CausalRM decomposes the model’s contextual embedding into two disentangled components: (1) causal factors that are sufficient for reward prediction, and (2) non-causal factors that capture reward-irrelevant attributes such as response length or stylistic bias. Building on this factorized representation, we constrain the reward model to predict rewards using only the causal factors, and optimize it with a standard pairwise preference loss augmented with mutual-information-based constraints. To further discourage the non-causal factors from carrying reward-predictive signals, we introduce an adversarial head trained to predict reward from these factors. Following 16 , we then optimize it with a gradient reversal layer (GRL), so that the adversarial head learns to predict reward while the encoder is pushed to remove reward-relevant information from the non-causal component. To summarize, our main contributions are three-fold: • We investigate the potential of causal representation learning for mitigating reward hacking in RLHF, and propose CausalRM , a framework that explicitly decomposes the latent representation into causal factors and reward-irrelevant non-causal factors. • To characterize both causal and non-causal representations, we design a novel VAE-based architecture with (1) a reward prediction head trained with a pairwise preference loss augmented by mutual information constraints, and (2) an adversarial head trained via gradient reversal to suppress reward-relevant signals in the non-causal component, which jointly enforces sufficiency, minimality, and causal invariance in the learned representations. • Extensive experiments on mathematical and dialogue tasks show that CausalRM improves both reward model accuracy and downstream RLHF performance. Furthermore, we demonstrate strong mitigation of reward hacking behaviors through reduced sensitivity to length and sycophantic bias, providing empirical validation of causal invariance in learned representations. 2 Preliminaries Reward modeling in RLHF. In RLHF, reward modeling aims to learn a scalar-valued function that approximates human preferences. Given a prompt x and a response y , a reward model outputs a scalar reward r_{\theta}(x,y) indicating the degree of preference for y under x . A widely used formulation is the Bradley–Terry model ( 6 ) , which defines the probability that a preferred response y^{w} is favored over a rejected response y^{l} as: p(y^{w}\succ y^{l}\mid x)\;=\;\frac{\exp(r_{\theta}(x,y^{w}))}{\exp(r_{\theta}(x,y^{w}))+\exp(r_{\theta}(x,y^{l}))}. (1) Given a human preference dataset \mathcal{D}=\{(x_{i},y_{i}^{w},y_{i}^{l})\}_{i=1}^{N} , the reward model is typically trained by minimizing the pairwise negative log-likelihood: \mathcal{L}_{\mathrm{RM}}(\theta)\;=\;-\mathbb{E}_{(x,y^{w},y^{l})\sim\mathcal{D}}\left[\log\sigma\!\left(r_{\theta}(x,y^{w})-r_{\theta}(x,y^{l})\right)\right], (2) where \sigma(\cdot) denotes the sigmoid function. Standard parameterization and reward hacking. In practice, the reward model is often initialized from a supervised fine-tuned (SFT) language model by reusing its backbone as a feature extractor and attaching a lightweight reward head, which is typically a single linear layer. Concretely, given a prompt–response pair (x,y) , the SFT backbone produces a contextual representation h\;=\;f_{\phi}(x,y), (3) and the reward head maps it to a scalar reward r_{\theta}(x,y)\;=\;g_{\psi}(h),\qquad\theta=(\phi,\psi), (4) where f_{\phi} denotes the pretrained backbone and g_{\psi} denotes the reward head. While effective, this parameterization can be vulnerable to reward hacking: the reward model may inherit and amplify preference-irrelevant signals already encoded in the SFT representation, causing the learned reward to correlate with spurious patterns rather than the causal drivers of human judgment ( 32 ; 56 ) . 3 Methodology In this section, we first analyze why standard reward models tend to learn spurious correlations from a causal perspective. Building on these insights, we construct a causal reward model, termed CausalRM, that filters out reward-irrelevant information via factored representation learning. Finally, we present the complete estimation procedure for CausalRM. 3.1 A Causal View of Reward Hacking The causal structure underlying standard reward modeling can be represented by Figure 1 , where

Read the complete text