Reviving The Classics: Active Reward Modeling in Large Language Model Alignment
Yunyi Shen, Hao Sun, Jean-François Ton · 2025
Read Reviving The Classics: Active Reward Modeling in Large Language Model Alignment by Yunyi Shen, Hao Sun, Jean-François Ton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04354.txt.
Introduction
Yunyi Shen, Hao Sun, and Jean-François Ton present a study from twenty twenty-five on picking the best pairs for human review in reward modeling. Reward models learn from human picks between two answers. Such picks are scarce and costly, so which pairs get picked matters a lot. The team draws on classic design theory from statistics.
A good pick set should span the space of cases and pair answers with a fair gap in worth. They use a Fisher based score to rank pairs for review, tuned to the last layer of the reward network. Tests on many models and data sets show strong scores, fast runs, and stable gains. Pairs across unlike prompts helped a lot, they found.
Catalog record
| CREATOR | Yunyi Shen, Hao Sun, Jean-François Ton |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04354
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reviving The Classics: Active Reward Modeling in Large Language Model Alignment Yunyi Shen, Hao Sun, Jean-François Ton Abstract Building neural reward models from human preferences is a pivotal component in reinforcement learning from human feedback (RLHF) and large language model alignment research. Given the scarcity and high cost of human annotation, how to select the most informative pairs to annotate is an essential yet challenging open problem. In this work, we highlight the insight that an ideal comparison dataset for reward modeling should balance exploration of the representation space and make informative comparisons between pairs with moderate reward differences. Technically, challenges arise in quantifying the two objectives and efficiently prioritizing the comparisons to be annotated. To address this, we propose the Fisher information-based selection strategies, adapt theories from the classical experimental design literature, and apply them to the final linear layer of the deep neural network-based reward modeling tasks. Empirically, our method demonstrates remarkable performance, high computational efficiency, and stability compared to other selection methods from deep learning and classical statistical literature across multiple open-source LLMs and datasets. Further ablation studies reveal that incorporating cross-prompt comparisons in active reward modeling significantly enhances labeling efficiency, shedding light on the potential for improved annotation strategies in RLHF. 1 Introduction The safe and successful deployment of Large Language Models (LLMs) across various application domains requires alignment with human values. Current research working on LLM alignment mainly focuses on reinforcement learning from human feedback (RLHF) ( Christiano et al.,, 2017 ; Stiennon et al.,, 2020 ; Ouyang et al.,, 2022 ; Bai et al., 2022a, ) , which rely on preference-based annotations provided by human annotators ( Bai et al., 2022b, ) . However, obtaining human feedback can be expensive, and the noisy and binary nature of such data often limits its information density, posing a challenge for effective reward modeling ( Wang et al., 2024a, ; Liu et al.,, 2024 ) . Active learning, where the model queries most informative labels based on its current state, offers a potential solution. It typically involves three key components: an initial model, a query strategy — often in the form of maximizing a scoring function over unlabeled data, and a pool of unlabeled data. The model selects a subset of data for labeling and retrains iteratively until a stopping criterion is met. In this work, we study the problem of active data acquisition in reward modeling. Technically, we introduce various scoring rules inspired by both classical experimental design ( Chaloner and Verdinelli,, 1995 ) and recent deep learning-based advancements ( Sener and Savarese,, 2017 ; Houlsby et al.,, 2011 ; Kirsch et al.,, 2019 ) . We adapt those methods to the learning of Bradley-Terry (BT) reward models ( Bradley and Terry,, 1952 ) , which have been successfully applied in large-scale alignment practices ( Ouyang et al.,, 2022 ; Touvron et al.,, 2023 ) and proven to be theoretically sound ( Sun et al.,, 2024 ) . We benchmark 8 scoring algorithms using 2 datasets and 3 LLMs, ranging in size from 2B to 8B, and evaluate a wide range of active learning setups. Our results show that two classical experimental design methods — applied to the final linear feature layer of deep models — achieve state-of-the-art performance and strong stability across different setups, model architectures, and datasets. Our main contributions can be summarized as follows: • Formally, we characterize the problem of optimal preference label annotation using embedding space BT regression framework and establish connections between active learning and classical experimental design literature under the BT context. • Methodologically, we introduce a set of algorithms inspired by classical experimental design literature, adapt them for deep BT regression models, and develop an efficient gradient approximation for the associated combinatorial optimization challenge in large-scale alignment problems. • Empirically, we evaluate different methods for preference label annotation across diverse setups, datasets, and base models. Our results suggest that applying classical experimental design techniques to the final layer of a deep neural network yields strong performance and stability. 2 Background and setup Reward modeling in alignment. Reinforcement learning is a key technique for aligning LLMs to ensure their safe and effective deployment ( Christiano et al.,, 2017 ; Ouyang et al.,, 2022 ; Stiennon et al.,, 2020 ) . The most prevailing approach, RLHF, relies on reward models as a fundamental mechanism for quantifying content quality and scaling the reinforcement learning ( Lambert et al.,, 2024 ; Wang et al., 2024a, ) . During fine-tuning and deployment, reward models serve as proxies for human evaluators ( Dubey et al.,, 2024 ; Dong et al.,, 2024 ; Wang et al., 2024b, ) , assessing how well LLM outputs align with human intent. Despite significant progress, reward modeling remains challenging due to the scarcity and inaccuracy of annotations ( Lambert et al.,, 2024 ; Wang et al., 2024a, ; Gao et al.,, 2023 ) . Prior research has attempted to mitigate these challenges through different aspects when learning from a fixed set of annotations ( Wang et al., 2024b, ; Winata et al.,, 2024 ; Liu et al.,, 2024 ; Lou et al.,, 2024 ; Coste et al.,, 2023 ; Zhang et al.,, 2024 ) . While Xiong et al., (2023) ; Dong et al., (2024) demonstrate that online annotations are more efficient in RLHF, the topic of online annotation prioritization strategy remain under-explored except for heuristic designs ( Muldrew et al.,, 2024 ) . Bradley-Terry model for reward modeling. A canonical model used for reward modeling from binary preference data is the Bradley-Terry (BT) model ( Bradley and Terry,, 1952 ) , or more precisely, its regression variant ( Sun et al.,, 2024 ) . In the most general setting, which allows for cross-prompt comparisons, a human annotator is presented with two pairs of prompts and responses, (x_{i,1},y_{i,1}) and (x_{i,2},y_{i,2}) . The annotator then provides a preference, h_{i}=1_{\{(x_{i,1},y_{i,1})\succ(x_{i,2},y_{i,2})\}} indicating whether the first pair is preferred over the second. Often, both responses correspond to the same prompt, i.e., x_{i,1}=x_{i,2} however, Bradley-Terry regression can operate without this assumption. The model regresses these annotations onto an embedding \bm{\Psi}(x_{i,1},y_{i,1}) . This is a mild assumption since these embeddings can be, for example, a concatenation of word embeddings, the output of tokenizers, or the output embedding of an LLM. When there is no risk of confusion, we denote the embeddings of pair i as \bm{\Psi}_{i,1}\in\mathbb{R}^{D} and \bm{\Psi}_{i,2}\in\mathbb{R}^{D} , with a reward function r\in\mathbb{R}^{D}\to\mathbb{R} . The goal is to learn this function from annotations. In the BT model, we assume that h_{i}\sim\BER\left(\sigma[r(\bm{\Psi}_{i,1})-r(\bm{\Psi}_{i,2})]\right) (1) with \sigma being the sigmoid function. Active learning. In a typical active learning setting, we have a labeled dataset, \mathcal{D}_{s}={(x_{i,1},y_{i,1},x_{i,2},y_{i,2},h_{i})}_{i=1}^{I_{s}} at step s , and a typically large pool of unlabeled data to be chosen from, \mathcal{P}_{s}={(x_{j,1},y_{j,1},x_{j,2},y_{j,2})} . The goal is to select a small subset \mathcal{C}_{s}\subset\mathcal{P}_{s} , subject to certain constraints, for labeling. Once labeled (denoted as \tilde{\mathcal{C}}_{s} ), this subset is added to the labeled dataset to train the next iteration of the model. We also consider this process in the embedding space, where the labeled and unlabeled sets are given by \mathcal{D}_{s}=\{(\bm{\Psi}_{i,1},\bm{\Psi}_{i,2},h_{i})\}_{i=1}^{I_{s}} and \mathcal{P}_{s}=\{(\bm{\Psi}_{j,1},\bm{\Psi}_{j,2})\}_{j=1}^{J_{s}} . A typical model-based active learning procedure is outlined in algorithm 1 . In this