Maximizing Signal in Human-Model Preference Alignment

Kelsey Kraus, Margaret Kroll · 2025

Read Maximizing Signal in Human-Model Preference Alignment by Kelsey Kraus, Margaret Kroll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04910.txt.

Introduction

Kelsey Kraus and Margaret Kroll argue that when people must agree with a model's calls, the model should be trained and judged on those people's own choices. Their 2025 paper looks at tasks like flagging toxic posts or picking the main points of a text. Quick automatic scores are cheap, they note, but they cannot replace human judgment where trust is at stake.

Much of the paper sorts signal from noise in human labels. Some disagreement is mere error and best practice can shrink it. Some of it is real, and it can teach a model what users in fact want. The authors close with a case study in which two safety filters were judged by human raters and tuned to match user taste. They offer the steps as a guide for teams that build language tools.

Catalog record

CREATORKelsey Kraus, Margaret Kroll
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04910

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Maximizing Signal in Human-Model Preference Alignment Kelsey Kraus, Margaret Kroll Abstract The emergence of powerful LLMs has led to a paradigm shift in Natural Language Understanding and Natural Language Generation. The properties that make LLMs so valuable for these tasks – creativity, ability to produce fluent speech, and ability to quickly and effectively abstract information from large corpora – also present new challenges to evaluating their outputs. The rush to market has led teams to fall back on quick, cost-effective automatic evaluations which offer value, but do not obviate the need for human judgments in model training and evaluation. This paper argues that in cases in which end users need to agree with the decisions made by ML models – e.g. in toxicity detection or extraction of main points for summarization – models should be trained and evaluated on data that represent the preferences of those users. We support this argument by explicating the role of human feedback in labeling and judgment tasks for model training and evaluation. First, we propose methods for disentangling noise from signal in labeling tasks. Then we show that noise in labeling disagreement can be minimized by adhering to proven methodological best practices, while signal can be maximized to play an integral role in model training and evaluation tasks. Finally, we illustrate best practices by providing a case study in which two guardrails classifiers are evaluated using human judgments to align final model behavior to user preferences. We aim for this paper to provide researchers and professionals with guidelines to integrating human judgments into their ML and generative AI evaluation toolkit, particularly when working toward achieving accurate and unbiased features that align with users’ needs and expectations. Introduction Since November 2022 with the launch of ChatGPT, the world has seen an explosion in popularity of generative AI content demonstrating impressive performance across a variety of tasks. This paradigm shift in Natural Language Generation (NLG) has changed the fields of text generation and summarization, computer-generated art, information retrieval, search engine optimization, and workflow automation. While new applications of generative technologies are transformative, research lags behind in effective, replicable strategies for evaluating their content. The core challenge of evaluating NLG systems lies in their very nature: models are prompted to generate human-like texts, and can respond in sophisticated, human-like, and nuanced ways. However, even with well-intentioned inputs, if left unsupervised generative AI output is prone to hallucinations and overconfident advice, and can produce output that fails to fulfill the goals of its human users ( Huang et al. 2023 ; Mahaut et al. 2024 ; Perković, Drobnjak, and Botički 2024 ) . Despite the rapid technological advancement in model performance, the field has not settled on a standardized way of evaluating LLM outputs that consistently aligns with human feedback. Traditional NLP evaluation methods remain popular with researchers, such as bleu ( Papineni et al. 2002 ) , rouge ( Lin 2004 ) , and meteor ( Lavie and Agarwal 2007 ) . These methods were originally developed as reference-based metrics for machine translation and are computationally cheap to run. However, research shows that the metrics have poor correlation with human judgments, and limited applicability across various aspects of NLG quality assessments ( Novikova et al. 2017 ; Reiter 2018 ; Howcroft et al. 2020 ) . More contemporary metrics, such as BERTScore, have also been shown in at least some applications to deviate widely from human preferences ( Zhang et al. 2019 ; Hanna and Bojar 2021 ) . SOTA advancements in self-evaluation techniques show promise in prompting LLMs to catch some of their own inconsistencies, as seen in Chain-of-Thought reasoning ( Liu et al. 2023 ) and LLM-as-Judge scenarios ( Zheng et al. 2020 ; Verga et al. 2024 ) ; however, these methods do not yet meet the bar set by human judges, nor do they show high performance on all dimensions of analysis ( Pavlovic and Poesio 2024 ; Baris Schlicht et al. 2024 ; Chen et al. 2024 ; Raina, Liusie, and Gales 2024 ; Thakur et al. 2024 ) . Despite the field’s push toward fast, automated metrics and unsupervised tasks, human judgments remain integral tools of model training and evaluation ( Gabriel and Ghazavi 2021 ) . Human labeling tasks arise when end users need to agree with the decisions made by ML models, e.g. in toxicity detection, image classification, or abstractive summarization. There are several methodological techniques that have been proposed for infusing human preference judgments into ML modeling. For example, RLHF allows the modeler to gather large-scale human preferences during model training, but is limited to a single-dimension preference paradigm ( Christiano et al. 2023 ) . The Constitutional AI framework forgoes directly gathering human preferences, relying on a self-supervised reward model where the AI trains itself using self-critiques and revisions ( Bai et al. 2022 ) . Such frameworks aim to provide methodological solutions to preference integration, but fail to answer core questions posed by the alignment community: Whose values and norms are being encoded in AI systems? And whose values and norms should these systems be aligned with? ( Bergman et al. 2024 ) Community-oriented frameworks for collecting diverse perspectives for AI alignment have made strides in advancing practices to ensure fairness and representation in AI ( Kirk et al. 2024 ) . However, these approaches are limited in their scalability due to their resource-intensiveness. This paper proposes a methodological framework to identify when resource-intensive alignment methods will offer the greatest return on investment. It also provides rigorous quantitative methods drawn from the social and biological sciences that can be used to evaluate the alignment performance of ML models. These methods help to fill a gap in the literature, which lacks a widely accepted standard for quantitatively analyzing human-preference data and for evaluating human-alignment performance of ML systems ( Card et al. 2020 ; van der Lee et al. 2021 ) . The paper is structured as follows. We first propose methods for disentangling noise from signal in labeling tasks. We show that noise in labeling disagreement can be minimized by adhering to proven methodological best practices, while signal in labeling disagreement can be maximized to play a foundational role in model training and evaluation. We then describe the analysis challenges posed by evaluating subjective content and propose concrete ways in which researchers can arbitrate disagreements that arise in labeled data. These suggestions can be integrated into strategies for generative AI content evaluation that promotes alignment between technical functionality and user-centric goals. This alignment is essential to promoting the development of trustworthy and effective AI systems. We conclude with a case study that exemplifies a model selection process supported by alignment with human feedback. Disagreement As Noise or Signal Most labeling tasks are subject to annotator disagreement. Even for seemingly objective or simple tasks, some inter-annotator disagreement will be observed due to unavoidable human limitations such as participant attention deficits, annotation guideline unclarities, and participant or administrator errors ( Reidsma and Carletta 2008 ) . In these cases, the disagreement can generally be resolved by identifying and fixing the mistakes, and/or by clarifying or improving the task methodology. We refer to this type of uninteresting disagreement as

Read the complete text