Beyond Single: A Data Selection Principle for LLM Alignment via Fine-Grained Preference Signals

Jia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu, Yi-Xuan Jin, Lan-Zhe Guo, Yu-Feng Li · 2025

Read Beyond Single: A Data Selection Principle for LLM Alignment via Fine-Grained Preference Signals by Jia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu, Yi-Xuan Jin, Lan-Zhe Guo, Yu-Feng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07638.txt.

Introduction

Jia Zhang and co-authors present a data selection rule for aligning language models with fine-grained human preferences, in a paper from 2025. People judge answers on many aspects at once, such as helpfulness and safety. Those aspects can clash, and methods trained on one blended score handle clashes poorly.

The team defines a measure of how much the aspects in a data set disagree. Rather than solve a hard joint optimization, they turn the task into picking the right training subset. Training on the examples with the most negative disagreement scores worked best in their tests. Theory supports the choice, and broad experiments show steady gains over training on all data. Good results came even with less than a third of the data.

Catalog record

CREATORJia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu, Yi-Xuan Jin, Lan-Zhe Guo, Yu-Feng Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Single: A Data Selection Principle for LLM Alignment via Fine-Grained Preference Signals Jia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu, Yi-Xuan Jin, Lan-Zhe Guo, Yu-Feng Li Abstract Aligning Large Language Models (LLMs) with diverse human values requires moving beyond a single holistic “better-than” preference criterion. While collecting fine-grained, aspect-specific preference data is more reliable and scalable, existing methods like Direct Preference Optimization (DPO) struggle with the severe noise and conflicts inherent in such aggregated datasets. In this paper, we tackle this challenge from a data-centric perspective. We first derive the Direct Multi-Preference Optimization (DMPO) objective, and uncover a key Preference Divergence (PD) term that quantifies inter-aspect preference conflicts. Instead of using this term for direct optimization, we leverage it to formulate a novel, theoretically-grounded data selection principle. Our principle advocates for selecting a subset of high-consensus data—identified by the most negative PD values—for efficient DPO training. We prove the optimality of this strategy by analyzing the loss bounds of the DMPO objective in the selection problem. To operationalize our approach, we introduce practical methods of PD term estimation and length bias mitigation, thereby proposing our PD selection method. Evaluation on the UltraFeedback dataset with three varying conflict levels shows that our simple yet effective strategy achieves over 10% relative improvement against both the standard holistic preference and a stronger oracle using aggregated preference signals, all while boosting training efficiency and obviating the need for intractable holistic preference annotating, unlocking the potential of robust LLM alignment via fine-grained preference signals. 1 Introduction Reinforcement Learning from Human Feedback (RLHF) ( Christiano et al. 2017 ; Ouyang et al. 2022 ; Bai et al. 2022 ) plays a pivotal role in aligning Large Language Models (LLMs) ( Naveed et al. 2024 ; Brown et al. 2020 ; Meta AI 2023 ) . However, standard RLHF methods of online reinforcement learning ( Schulman et al. 2017 ; Shao et al. 2024 ) are often plagued by substantial computational overhead and the complex, multi-stage training process. As an efficient alternative, Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) directly aligns LLMs by fine-tuning them on an offline dataset of human preferences, bypassing the need for the complex and unstable RL-based methods. The efficacy of DPO is tied to the quality of the offline preference dataset. Typically, practitioners collect and annotate data by soliciting judgments from human annotators or stronger teacher models ( Ziegler et al. 2020 ; Stiennon et al. 2020 ; Bai et al. 2022 ) to label a winner from two responses for a given prompt. However, this paradigm rests on the implicit but often untenable assumption that a single, holistic, and tractable criterion for preference annotating exists. In many cases, it is often ambiguous which of two candidates is definitively superior under a holistic criterion ( Bakker et al. 2022 ; Casper et al. 2023 ) . Specifically, the ambiguity stems from two primary issues. 1) Defining a single, comprehensive criterion is inherently challenging, causing annotators with diverse biases to struggle with a consistent judgment. 2) Responses often exhibit quality trade-offs, where one is rarely superior across all aspects. This complexity makes it intractable for annotators to provide reliable judgments and annotations. Consequently, the resulting inconsistent and unreliable preference signals risk undermining the effectiveness of alignment methods. Instead of relying on a holistic, ambiguous “better-than” judgment, some studies ( Ji et al. 2023 ; Wu et al. 2023 ; Rame et al. 2023 ) argue that the preference can be decomposed into multiple, fine-grained aspects, which we term sub-preferences for simplicity. For example, judging if a response is superior in terms of logical consistency, factual accuracy, or instructional clarity is more tractable than making a holistic quality assessment. Such fine-grained criteria are much easier to define precisely, facilitating annotators to provide more reliable and consistent preference labels, as annotators can then focus on one clear aspect at a time. Leveraging aggregated datasets sourced from various sub-preference criteria appears promising for alignment, given their ease of acquisition and higher reliability. However, this path faces two hurdles. 1) Existing DPO-style methods are mainly designed for a single preference criterion and lack mechanisms to handle multiple fine-grained preference signals. 2) More critically, aggregating data from diverse sub-preferences introduces severe data quality issues, such as redundancy, noise, and preference conflicts, leading to training inefficiency and performance degradation. For instance, the statistical analysis in Figure 1 of the widely-used preference dataset UltraFeedback ( Cui et al. 2023 ) , which provides annotations of fine-grained aspects, reveals that nearly 30% of samples exhibit explicit conflict between fine-grained and holistic preferences. This predicament raises a critical research question: How can we effectively harness these aggregated datasets with fine-grained preference signals for robust model alignment, especially in the presence of inherent and severe preference conflicts and noise? In this paper, we explore the understudied alignment scenario of using fine-grained preference data and signals from a data-centric perspective. We begin by deriving the objective for Direct Multi-Preference Optimization (DMPO) that utilizes all signals for alignment. Inspired by the insight that the DMPO assigns varying importance to preference data, we formulate a theoretically-grounded data selection principle based on preference divergence by proving the optimal loss bounds of DMPO in the selection problem. To facilitate the usability in practice, we propose PD term estimation and length bias mitigation, resulting in our final simple but effective data selection algorithm for LLM alignment. We evaluated our method on the UltraFeedback dataset under various preference conflict scenarios. Leveraging only fine-grained preference signals, our approach consistently outperforms baselines trained on both holistic and even stronger oracle preference, while simultaneously boosting training efficiency and obviating the need for intractable holistic preference annotating. This demonstrates the effectiveness of our strategy in filtering high-quality data from datasets with noise and conflicts, enabling more robust and efficient alignment using fine-grained preference signals. To summarize, our contributions are as follows: (1) Selection Problem : To address data issues such as noises and conflicts in alignment using fine-grained preference signals, we formalize the data selection problem grounded in insights from the derived DMPO objective. (2) Principled Method : We propose a novel, theoretically-grounded data selection principle: train only on a subset of high-consensus data identified by the most negative PD values. To operationalize this, we introduce practical methods of PD term estimation and length bias mitigation, resulting in our PD selection method. (3) Empirical Validation : Extensive experiments show that our selection method outperforms the standard holistic preference and a stronger oracle baseline across varying conflict levels, demonstrating its effectiveness in filtering for high-quality data from datasets with fine-grained preference signals for LLM alignment. 2 Related Work 2.1 Preference Alignment Preference alignment is crucial for ensuring that the behavior of LLMs conforms to the expectations of human values. Prominent alignment methods include supervised fine-tuning (SFT) ( Wei et al. 2022 ; Ouyang et al. 2022 ) , reinforcement learning fine-tuning (RLFT) ( Schulman et al. 2017 ; Shao et al. 2024 ; Hu et al. 2025 ) , and DPO-style approaches ( Rafailov et al. 2023 ; Azar et al. 2024 ; Ethayarajh et al. 2024 ; Meng, Xia, and Chen 2024 ) which directly optimize the model on preference

Read the complete text