Scalable Valuation of Human Feedback through Provably Robust Model Alignment

Masahiro Fujisawa, Masaki Adachi, Michael A. Osborne · 2025

Read Scalable Valuation of Human Feedback through Provably Robust Model Alignment by Masahiro Fujisawa, Masaki Adachi, Michael A. Osborne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17859.txt.

Introduction

Masahiro Fujisawa, Masaki Adachi, and Michael Osborne tackle noisy human feedback in AI training in a 2025 paper. Crowd feedback often favors worse replies, which skews models trained to please raters. A truly robust aim should give the same model even under heavy label noise.

The authors prove no current alignment method holds that strong property, called redescending. Their new loss is the first with a proof of that trait in hand. The trained model then gauges how clean each feedback set is and where its bad labels sit. That gauge needs no gradients, so it scales to huge sets at low cost. On a known chat data set it found deep noise, and cleaning it lifted all methods.

Catalog record

CREATORMasahiro Fujisawa, Masaki Adachi, Michael A. Osborne
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE131 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17859

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable Valuation of Human Feedback through Provably Robust Model Alignment Masahiro Fujisawa, Masaki Adachi, Michael A. Osborne Abstract Despite the importance of aligning language models with human preferences, crowd-sourced human feedback is often noisy—for example, preferring less desirable responses—posing a fundamental challenge to alignment. A truly robust alignment objective should yield identical model parameters even under severe label noise, a property known as redescending . We prove that no existing alignment methods satisfy this property. To address this, we propose Hölder-DPO, the first principled alignment loss with a provable redescending property, enabling estimation of the clean data distribution from noisy feedback. The aligned model estimates the likelihood of clean data, providing a theoretically grounded metric for dataset valuation that identifies the location and fraction of mislabels. This metric is gradient-free, enabling scalable and automated human feedback valuation without costly manual verification or clean validation dataset. Hölder-DPO achieves state-of-the-art robust alignment performance while accurately detecting mislabels in controlled datasets. Finally, we apply Hölder-DPO to widely used alignment datasets, revealing substantial noise levels and demonstrating that removing these mislabels significantly improves alignment performance across methods. 1 Introduction Aligning large language models (LLMs; [ 1 , 93 , 90 ] ) with human preferences is essential for value alignment and mitigating safety risks [ 63 , 9 ] . Direct Preference Optimization (DPO; [ 82 ] ) has emerged as a key method, fine-tuning LLMs using pairwise comparisons—e.g., between preferred and rejected responses [ 24 , 17 ] . This framework supports alignment with a broad range of rankable values, such as helpfulness [ 8 ] , toxicity reduction [ 50 ] , and summarization quality [ 88 , 95 ] . Human feedback is typically gathered via crowdsourcing, including from platforms like ChatGPT [ 1 ] , but is often noisy—e.g., preferring undesirable responses—which can significantly deteriorate model performance [ 103 , 23 , 28 ] . In fact, the Anthropic HH-RLHF dataset [ 33 ] reportedly contain over 25% inconsistent feedback [ 86 , 19 , 103 ] . Plus, recent work [ 106 , 41 , 15 ] shows that models trained on smaller, high-quality datasets can outperform those trained on larger, noisier ones. However, maintaining data quality often requires costly manual review (e.g., inter-annotator agreement), highlighting the need for scalable, automated methods to assess and clean human feedback. To mitigate the impact of noisy human feedback, researchers have proposed robustifed DPO variants. Among various heuristics [ 73 , 6 , 98 ] , two methods claim provable guarantees: Provably Robust DPO (R-DPO) [ 23 ] and Distributionally Robust DPO (Dr. DPO) [ 103 ] . However, their notion of robustness is limited to bounding the parameter estimation error—bounds that degrade as the contamination ratio \epsilon (the fraction of mislabeled data) increases. In contrast, truly robust methods should estimate the clean data distribution precisely under heavy contamination. This stronger form of robustness corresponds to the redescending property [ 71 ] , a classical concept in robust statistics formalized via the influence function (IF; [ 44 , 26 ] ). We analyze existing (robust) DPO variants through IFs and prove that none satisfy the redescending property. If this property holds, the aligned model approximates the clean data distribution—enabling principled mislabel detection, as mislabels naturally exhibit lower likelihoods being clean data points. Crucially, identifying mislabeled data has benefits beyond robustness: it supports annotator evaluation, incentivizes higher-quality feedback, and enables scalable dataset cleaning, reducing reliance on costly manual verification and promoting more transparent, interpretable training pipelines. To address this challenge, we propose Hölder-DPO, the first DPO variant with a provable redescending property. As illustrated in Figure 1 , fine-tuning an LLM with Hölder-DPO not only yields a robustly aligned model but also enables the model itself to serve as a mislabel valuation metric. Unlike existing data valuation methods that rely on direct IF computation [ 58 , 21 ] , Hölder-DPO requires only likelihood evaluation—bypassing costly gradient-based approximations and enabling scalable dataset valuation. Hölder-DPO demonstrates superior robustness and accurately detects mislabeled examples across datasets. Applied to the Anthropic HH dataset [ 33 ] , it reveals substantial noise, and removing the identified mislabels significantly improves alignment performance across methods. Figure 1: While existing DPO variants are vulnerable to \epsilon -contamination, Hölder-DPO is provably robust. It also ranks data points by clean-data likelihood, enabling mislabel identification. 2 Preliminaries 2.1 Language Model Alignment by Direct Preferential Optimization Human feedback and Bradley-Terry model. Let x\in\mathcal{X} be a prompt sampled from a distribution p(x) over a finite context space \mathcal{X} . The policy model \pi(y\mid x) generates a pair of responses (y_{1},y_{2}) . A human annotator then provides a preference between them, identifying the preferred response as y_{\textrm{win}} , and the rejected response as y_{\textrm{lose}} , i.e., y_{\textrm{win}}\succ y_{\textrm{lose}} . Although the underlying reward function r^{*}(x,y) , which ranks responses, is unobservable, Bradley–Terry (BT) model [ 12 ] provides a standard framework for modeling pairwise comparisons based solely on observed preferences: \displaystyle p^{*}(y_{1}\succ y_{2}\mid x,y_{1},y_{2})\coloneqq\sigma\left[\exp(r^{*}(x,y_{1}))-\exp(r^{*}(x,y_{2}))\right], (1) where \sigma(a)\coloneqq 1/(1+\exp(-a)) is the logistic function. For notational convenience, we define a pairwise data point as s\coloneqq\{x,y_{\textrm{win}},y_{\textrm{lose}}\} , and denote the preference dataset as \mathcal{D}\coloneqq\{s^{(i)}\}_{i=1}^{N} . Each data point s^{(i)} is sampled independently from p_{\mathcal{D}}(s) , which is the following joint distribution p_{\mathcal{D}}(s)\propto p^{*}(y_{\text{win}}\succ y_{\text{lose}}\mid x,y_{1},y_{2})\cdot\pi(y_{1},y_{2}\mid x)\cdot p(x) . Direct Preference Optimization (DPO). The goal of preference-based model alignment is to find parameters \theta such that the policy \pi_{\theta} accurately approximates either the latent reward function r^{*} or the preference distribution p^{*} . To achieve this, Rafailov et al. 2023 proposed DPO, which efficiently solves the alignment task by minimizing the following optimization problem \mathcal{L}(s,\pi_{\theta}) : \displaystyle\argmin_{\theta}\mathbb{E}_{p_{\mathcal{D}}(s)}[-\log\sigma(g_{\theta}(s))],\,\,\text{where}\,\,g_{\theta}(s)=\beta\log\frac{\pi_{\theta}(y_{\textrm{win}}\mid x)}{\pi_{\mathrm{ref}}(y_{\textrm{win}}\mid x)}-\beta\log\frac{\pi_{\theta}(y_{\textrm{lose}}\mid x)}{\pi_{\mathrm{ref}}(y_{\textrm{lose}}\mid x)}, (2) where \pi_{\mathrm{ref}} is the reference model, and \beta>0 is a hyperparameter controlling alignment strength. The reference model is typically obtained by supervised fine-tuning (SFT) on the prompts x with the preferred response y_{\text{win}} only [ 82 , 84 ] . The corresponding gradient is given by: \displaystyle\nabla_{\theta}\mathcal{L}(s,\pi_{\theta}) \displaystyle\coloneqq\sigma(-g_{\theta}(s))\cdot\left(\nabla_{\theta}\log\pi_{\theta}(y_{\textrm{win}}\mid x)-\nabla_{\theta}\log\pi_{\theta}(y_{\textrm{lose}}\mid x)\right), (3) where g_{\theta}(s)\coloneqq\hat{r}_{\theta}(x,y_{\textrm{win}})-\hat{r}_{\theta}(x,y_{\textrm{lose}}) and \hat{r}_{\theta}(x,y)=\beta\log(\nicefrac{{\pi_{\theta}(y\mid x)}}{{\pi_{\mathrm{ref}}(y\mid x)}}) denotes the implicit reward function induced by \pi_{\theta} . 2.2 Data Contamination Model for Label Flips and Redescending Property Data contamination model. Noise in human preference datasets [ 34 , 103 , 97 ] can arise from two sources: noise in the responses (y_{1},y_{2}) [ 42 , 103 ] , such as irrelevant or incoherent samples, and noise in preference feedback y_{\text{win}}\succ y_{\text{lose}} [ 34 , 85 , 27 , 22 , 103 ] , due to incorrect preference annotations (also known as label flips). As Wu et al. 2024a proved that vanilla DPO is robust against noise in responses, we focuses on robustness to label flips. We model the label flip process as follows: We denote a flipped data point as s_{\textrm{flip}}\coloneqq\{x,y_{\mathrm{win}}^{\textrm{flip}},y_{\mathrm{lose}}^{\textrm{flip}}\} , where y_{\mathrm{win}}^{\textrm{flip}}=y_{\mathrm{lose}} and y_{\mathrm{lose}}^{\textrm{flip}}=y_{\mathrm{win}} represent a flipped preference label pair. Such a flipped point s_{\textrm{flip}} is sampled from the distribution p(s_{\textrm{flip}})\propto p_{\textrm{flip}}(y_{\mathrm{win}}^{\textrm{flip}}\succ y_{\mathrm{lose}}^{\textrm{flip}}\mid x,y_{1},y_{2})\cdot\pi(y_{1},y_{2}\mid x)\cdot p(x) . Contaminated dataset \widetilde{\mathcal{D}} is given by: Definition 1 ( \epsilon -contamination model ) . Let 0\leq\epsilon<1/2 be contamination ratio. The generative distribution of \widetilde{\mathcal{D}} under the \epsilon -contamination model is defined as p^{(\epsilon)}_{\widetilde{\mathcal{D}}}(\tilde{s})=(1-\epsilon)p_{\mathcal{D}}(s)+\epsilon p(s_{\textrm{flip}}) , and denote \tilde{s} as a contaminated data point, which can be either clean s or

Read the complete text