Learning Guarantee of Reward Modeling Using Deep Neural Networks
Yuanhang Luo \rm{Luo}, Yeheng Ge \rm{Ge}, Ruijian Han \rm{Han}, Guohao Shen \rm{Shen}, Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Department of Applied Mathematics, The Hong Kong Polytechnic University · 2025
Read Learning Guarantee of Reward Modeling Using Deep Neural Networks by Yuanhang Luo \rm{Luo}, Yeheng Ge \rm{Ge}, Ruijian Han \rm{Han}, Guohao Shen \rm{Shen}, Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Department of Applied Mathematics, The Hong Kong Polytechnic University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06601.txt.
Introduction
Yuanhang Luo and co-authors study, in a 2025 paper, how well a deep neural network can learn a reward score from pairs of choices. Such scores guide models trained from human feedback: people pick the better of two replies, and the model learns what to aim for. The team asks how much error such a learner can be held to, in a setting where no fixed formula is assumed for the reward.
They prove a fresh bound on that error, one that depends in a clear way on the shape of the network used. They then add a key premise: the best act wins its pairs by a clear margin, not by a hair. Under that premise the bound gets much sharper. The result helps explain why clear human picks make feedback training work well.
Catalog record
| CREATOR | Yuanhang Luo \rm{Luo}, Yeheng Ge \rm{Ge}, Ruijian Han \rm{Han}, Guohao Shen \rm{Shen}, Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Department of Applied Mathematics, The Hong Kong Polytechnic University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Guarantee of Reward Modeling Using Deep Neural Networks Yuanhang Luo \rm{Luo}, Yeheng Ge \rm{Ge}, Ruijian Han \rm{Han}, Guohao Shen \rm{Shen}, Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Department of Applied Mathematics, The Hong Kong Polytechnic University Abstract In this work, we study the learning theory of reward modeling with pairwise comparison data using deep neural networks. We establish a novel non-asymptotic regret bound for deep reward estimators in a non-parametric setting, which depends explicitly on the network architecture. Furthermore, to underscore the critical importance of clear human beliefs, we introduce a margin-type condition that assumes the conditional winning probability of the optimal action in pairwise comparisons is significantly distanced from 1/2. This condition enables a sharper regret bound, which substantiates the empirical efficiency of Reinforcement Learning from Human Feedback and highlights clear human beliefs in its success. Notably, this improvement stems from high-quality pairwise comparison data implied by the margin-type condition, is independent of the specific estimators used, and thus applies to various learning algorithms and models. 1 Introduction Reinforcement Learning from Human Feedback (RLHF) has proven highly effective in aligning large language models with human preferences and expert policies ( Christiano et al., 2017 ) . A significant advancement in this domain is Direct Preference Optimization (DPO), which enhances RLHF by learning directly from pairwise comparison data, bypassing the need for an iterative reinforcement learning fine-tuning pipeline. This method not only improves efficiency but also strengthens alignment with human preferences ( Rafailov et al., 2024 ) , particularly in contexts with intuitively clear feedback, such as recommendation systems and image generation. Recent studies have explored the theoretical underpinnings of RLHF, yet performance guarantees remain an open challenge. Zhu et al., 2023 investigated the theoretical properties of reward modeling based on action-based comparisons, while Chen et al., 2022 and Saha et al., 2023 focused on trajectory-based comparisons. However, these analyses have largely focused on specific functional classes with limited representational power, neglecting bias due to approximation errors under misspecification. Despite the success of deep neural networks (DNNs) in RLHF due to their expressive power, their approximation properties and estimation bias remain theoretically underexplored. This gap in the literature highlights the necessity for a more comprehensive theoretical framework to fully leverage the potential of DNN estimators in RLHF. Sample efficiency is a fundamental theoretical challenge in RLHF. Unlike traditional RL, which often requires vast amounts of environmental interaction, RLHF can achieve strong performance with relatively few pairwise comparison data ( Christiano et al., 2017 ) . This efficiency gain likely stems from the informational richness of human preferences: by explicitly ranking actions or outputs, human feedback provides a dense learning signal that reduces the need for exploration ( Wang et al., 2024 ) . However, this advantage depends critically on the quality of preferences. Ambiguous or conflicting feedback can obscure the true reward function, while high-consensus datasets, such as those used to fine-tune InstructGPT ( Ouyang et al., 2022 ) , enable robust reward modeling. Existing RLHF studies emphasize the empirical benefits of unambiguous feedback, but theoretical quantification of how such clarity translates to sample efficiency remains a critical gap. To address these challenges, this paper offers non-parametric theoretical guarantees for DNN-based reward modeling and develops a mathematical model to calibrate clear preferences in RLHF, enhancing our understanding of its sample efficiency. Specifically, our contributions focus on the following aspects: • Regret Bounds of DNN Estimators. We establish regret bounds for reward modeling utilizing deep neural networks with pairwise comparison data. These bounds balance approximation and stochastic error based on the sample size, providing critical insights into the efficacy of DNN-based estimators in RLHF and underscoring their sample efficiency. • Calibration of Clear Human Beliefs. We introduce a novel margin-type condition to calibrate clear human beliefs in RLHF. This condition suggests high-quality pairwise comparison datasets and reveals the structure of the underlying reward, leading to sharper regret bounds. Our findings emphasize the importance of clear human beliefs in the success of RLHF, with these theoretical improvements being applicable across diverse learning algorithms. • Empirical Investigations. We demonstrate the necessity for high-quality pairwise comparison data and validate our theory through experiments with varying DNN architectures, emphasizing the broad applicability of our theoretical results in reward modeling. The paper is organized as follows: Section 2 introduces the pairwise comparison model and a margin-type condition that accelerates regret convergence. Section 3 derives architecture-dependent non-asymptotic regret bounds and their implications for deep reward estimators. Numerical experiments in Section 4 examine how network architecture and comparison data quality affect estimation performance. Sections 5 - 6 provide a literature review and conclusion. Technical details are deferred to the Appendix. 2 Pairwise Comparison, Margin-type Condition and Sharper Bound In this work, we consider the reward modeling in the action-based pairwise comparison case. Let \mathcal{S} be the set of states (prompts), and \mathcal{A} be the set of actions (responses). We consider a pairwise comparison dataset \{s^{i},a^{i}_{1},a^{i}_{0},y^{i}\}^{N}_{i=1} with sample size N : the state s^{i} is sampled from the probability measure on state space \mathcal{S} , denoted by \rho_{s} ; Conditioning on the state s^{i} , the action pair (a^{i}_{1},a^{i}_{0}) is sampled from some joint distribution \mathbb{P}(a_{1},a_{0}|s^{i}) ; The comparison outcome y^{i} indicates the preference between a^{i}_{1} and a^{i}_{0} . Specifically, a^{i}_{1} is preferred over a^{i}_{0} if y^{i}>0 and conversely, a^{i}_{0} is preferred if y^{i}<0 . It is worth noting that we do not restrict the outcome to a binary format, and it accommodates various types of outcomes discussed in the literature. For simplicity, readers may consider the binary case where y^{i} takes on values of either -1 or 1 . We define the reward function r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} , which evaluates the reward of taking each action at a given state. We denote d as the dimension of the input for the reward function r . For any reward function r , we denote the decision maker as \pi_{r}(s)=\argmax_{a\in\mathcal{A}}\ r(s,a) . Let r^{*}(s,a) denote the underlying true reward function. We define the optimal action for the state s by \pi_{r^{*}}(s)=\argmax_{a\in\mathcal{A}}\ r^{*}(s,a) . For an estimated reward function r , we are interested in the regret of the induced \pi_{r}(s) , which is \mathcal{E}(r)=\int_{\mathcal{S}}r^{*}(s,\pi_{r^{*}}(s))-r^{*}(s,\pi_{r}(s))d\rho_{s}. (1) The regret ( 1 ) is an intrinsic measure for evaluating RLHF ( Zhu et al., 2023 ; Zhan et al., 2024 ) . It is important to note that the policy \pi_{r} is induced from the reward r and the resulting regret is determined by r . In practice, we optimize the comparison model using crowd-sourced comparison outcomes. Even if the reward function is not perfectly estimated, there remains an opportunity to derive a correct policy and achieve low regret in reinforcement learning tasks. This potential stems from clear human beliefs, which suggest a significant gap between the reward of the optimal action and its alternatives. However, the critical role of reward differences between actions is often overlooked in pairwise comparison analysis, which typically relies on the smoothness of the reward function. In light of this, we are motivated to quantify the effects of reward differences on regret, capturing the reward gap between actions. As discussed in Wang et al., 2024 ; Song et al., 2024 ; Zhan et al.,