Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation

Jianxiang Zang, Meiling Ning, Shihan Dou, Jiazheng Zhang, Tao Gui, Qi Zhang, Xuanjing Huang · 2025

Read Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation by Jianxiang Zang, Meiling Ning, Shihan Dou, Jiazheng Zhang, Tao Gui, Qi Zhang, Xuanjing Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02618.txt.

Introduction

Jianxiang Zang and co-authors tackle a flaw in how reward models judge language model answers, in a 2025 paper. Reward models steer training from human feedback. Their judgments can be gamed when attention lands on the wrong words. The authors trace the flaw to how these models read text in one direction only, and to how answer pairs are read apart.

Their fix trains the reward model to copy the attention patterns of a stronger reading model. That teacher reads text with full, two-way attention. An alignment goal guides the reward model toward those patterns. Tests showed steadier and more general reward signals than rival fixes. Rival methods mostly targeted noisy training data instead. The authors argue the attention flaw is the deeper limit. Fixing it should make feedback training more robust.

Catalog record

CREATORJianxiang Zang, Meiling Ning, Shihan Dou, Jiazheng Zhang, Tao Gui, Qi Zhang, Xuanjing Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02618

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation Jianxiang Zang, Meiling Ning, Shihan Dou, Jiazheng Zhang, Tao Gui, Qi Zhang, Xuanjing Huang Abstract The reward model (RM), as the core component of reinforcement learning from human feedback (RLHF) for large language models (LLMs), responsible for providing reward signals to generated responses. However, mainstream preference modeling in RM is inadequate in terms of token-level interaction, making its judgment signals vulnerable to being hacked by misallocated attention to context. This stems from two fundamental limitations: (1) Current preference modeling employs decoder-only architectures, where the unidirectional causal attention mechanism leads to forward-decaying intra-sequence attention within the prompt-response sequence. (2) The independent Siamese-encoding paradigm induces the absence of token-level inter-sequence attention between chosen and rejected sequences. To address this “attention hacking”, we propose “ Interaction Distillation ”, a novel training framework for more adequate preference modeling through attention-level optimization. The method introduces an interaction-based natural language understanding model as the teacher to provide sophisticated token interaction patterns via comprehensive attention, and guides the preference modeling to simulate teacher model’s interaction pattern through an attentional alignment objective. Through extensive experiments, interaction distillation has demonstrated its ability to provide more stable and generalizable reward signals compared to state-of-the-art RM optimization methods that target data noise, highlighting the attention hacking constitute a more fundamental limitation in RM. Introduction Recently, reinforcement learning from human feedback (RLHF) has been widely adopted to enhance the alignment between large language model (LLM) behaviors and human preferences ( Ouyang et al. 2022 ; Shen et al. 2023 ; Xu et al. 2025 ) . The reward model (RM), as the core component of RLHF, is responsible for providing reward signals for LLM-generated responses ( Wang et al. 2024a ; Dou et al. 2024 ; Zhong et al. 2025 ) , guiding LLMs to produce responses with higher reward scores. The current RM training approach first concatenates the prompt and response into a sequence, uses a LLM as the backbone to obtain sequence representations, and maps them to a scalar reward score via a linear layer. It then enforces the Bradley-Terry-based preference modeling ( Bradley and Terry 1952 ) constraint to ensure that the chosen response receives a higher reward score than the rejected one ( Schulman et al. 2017 ; Wang et al. 2024a ) . However, this preference modeling is inadequate at the token interaction level, as both intra-sequence and inter-sequence attention remain incomplete during training. This incompleteness makes the reward signals vulnerable to being hacked by misallocated attention to context. This “ attention hacking ” stems from two interaction limitations: (1) From intra-sequence attention perspective, existing preference-based RMs rely entirely on Transformer decoders as their backbone, where sequence attention is implemented via unidirectional causal multi-head attention . This forward-decaying attention mechanism prevents the RM from capturing interactions with preceding tokens in the sequence, resulting in disproportionate focus on the latter tokens of the prompt-response sequence. (2) From inter-sequence attention perspective, the independent Siamese-encoding paradigm in preference modeling leads to the absence of early interaction attention between chosen and rejected sequences. This design flaw prevents the RM from capturing token-level interactions, resulting in inadequate cross-sequence semantic perception. These token-level interaction limitations led us directly to consider the interaction-based natural language understanding (NLU) model , which is based on fine-tuned encoder-only Transformer (the BERT family ( Devlin et al. 2019 ; He et al. 2022 ) ) on NLU datasets ( Camburu et al. 2018 ) . It performs downstream classification tasks by jointly encoding the concatenation of two sequences. Since it processes the concatenated sequence, its bidirectional and global attention models the most comprehensive interactions between intra-sequence and inter-sequence tokens. Therefore, although interaction-based NLU models with the BERT family as their backbone are significantly smaller in scale compared to RMs based on LLMs, they serve as stronger teacher models in terms of attention. Inspired by this, we propose “ Interaction Distillation ”, a novel framework aiming to achieve more adequate token-level interaction in preference-based reward modeling, and the reward model trained with this method is termed Id-Rm . This framework consists of three steps: (1) Introducing an interaction-based NLU model as the teacher model, which unified encodes concatenated sequence pairs. Since it operates on concatenated sequences, its MHA can fully model both intra-sequence and inter-sequence token interactions. Consequently, we capture its global attention map as the learning objective for RM. (2) Leveraging the query and key matrices from preference modeling to simulate the teacher model’s comprehensive attention computation, guiding the standard RM to simulate the sophisticated interaction patterns of the teacher model. (3) Designing an attentional optimization objective to align the simulated attention maps from preference modeling with the real and comprehensive attention maps from the interaction-based NLU model, thereby bridging the gap in token-level interaction between RM and the teacher model. The current state-of-the-art (SOTA) preference-based reward modeling optimization methods primarily address data noise ( Touvron et al. 2023 ; Wang et al. 2024b ; Rame et al. 2024 ; Miao et al. 2024 ) . In contrast, Id-Rm delivers more stable and generalizable reward signals without introducing inference costs, with its advantages manifested in two key aspects: (1) For RLHF tasks ( Völske et al. 2017 ; Bai et al. 2022 ) , the Id-Rm -optimized policy models generate responses that best align with human preferences. (2) In out-of-distribution preference perception tasks ( Lambert et al. 2024 ; Zhou et al. 2024 ) , Id-Rm achieves the highest average accuracy. These results highlight that the intrinsic deficiencies in RM’s attention mechanisms constitute a more fundamental limiting factor than data noise. In conclusion, our main contributions are summarized as follows: • We systematically analyze the attention hacking in reward modeling, which stems from inadequate token-level interactions: (1) forward-decaying intra-sequence attention caused by unidirectional causal attention; (2) absent token-level inter-sequence attention due to the independent Siamese-encoding paradigm. • We propose interaction distillation, a novel preference modeling method. This method introduces an interaction-based NLU model as the teacher to provide a comprehensive attention map, and guides the preference modeling to simulate this sophisticated interaction pattern through an attentional distillation objective. • Through extensive experiments on both RLHF and OOD preference perception tasks, interaction distillation demonstrates its ability to deliver more stable and generalizable reward signals compared to other SOTA reward modeling optimization approaches, without introducing inference costs. Attention Hacking in Reward Modeling In this section, we first introduce the training objective for preference-based reward models. We then analyze how both intra- and inter-sequence attention in preference modeling are directly linked to the reward model’s decision. However, the inherent nature of preference modeling is inadequate at the token interaction level, making reward signals vulnerable to being hacked by misallocated attention to context. Figure 1: Attentional analysis of the reward model trained on the HH-RLHF dataset on the test set . The line graph shows the average accuracy of the reward model on the test set when masking content at different stages; the bar chart represents the normalized average cumulative attention at each stage from the beginning to the end of the joint prompt-response sequence, where “correct”, “incorrect” denote correctly classified and misclassified sequences. Preference-based Reward Modeling Generally, the \theta parameterized reward

Read the complete text