Understanding Impact of Human Feedback via Influence Functions
Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee, [1ex] KAIST; Columbia University[1ex] [email protected] · 2025
Read Understanding Impact of Human Feedback via Influence Functions by Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee, [1ex] KAIST; Columbia University[1ex] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05790.txt.
Introduction
Taywon Min and three co-authors use in 2025 influence functions to measure how single pieces of human feedback shape the reward models that steer large language models. Feedback from labelers can be noisy, inconsistent, or biased, and bad rewards quietly misalign the final model. Knowing which labels did the damage is the first step to fixing them.
A cheaper approximation makes influence scores practical for large models and big preference datasets. Two uses are demonstrated: spotting common labeler biases hiding in feedback data, and coaching labelers to align better with expert judgment. The work treats feedback itself as data to audit, a step toward oversight that scales with the models being overseen.
Catalog record
| CREATOR | Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee, [1ex] KAIST; Columbia University[1ex] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05790
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Impact of Human Feedback via Influence Functions Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee, [1ex] KAIST; Columbia University[1ex] [email protected] Abstract In Reinforcement Learning from Human Feedback (RLHF), it is crucial to learn suitable reward models from human feedback to align large language models (LLMs) with human intentions. However, human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses. Such feedback can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process. To address these challenges, we explore the use of influence functions to measure the impact of human feedback on the performance of reward models. We propose a compute-efficient approximation method that enables the application of influence functions to LLM-based reward models and large-scale preference datasets. Our experiments showcase two key applications of influence functions: (1) detecting common labeler biases in human feedback datasets and (2) guiding labelers in refining their strategies to better align with expert feedback. By quantifying the impact of human feedback, we believe that influence functions can enhance feedback interpretability and contribute to scalable oversight in RLHF, helping labelers provide more accurate and consistent feedback. Source code is available at https://github.com/mintaywon/IF_RLHF . 1 Introduction As large language models (LLMs) demonstrate remarkable capabilities across various domains, ensuring their behaviors align with human intentions becomes increasingly important. To this end, reinforcement learning from human feedback (RLHF) has emerged as a powerful solution for fine-tuning LLMs ( Ziegler et al. 2019 ; Stiennon et al. 2020 ; Ouyang et al. 2022 ) . In RLHF, human feedback is collected to train reward models that capture important human values, such as helpfulness and harmlessness ( Bai et al. 2022 ; Ji et al. 2024 ) . LLMs are then fine-tuned to produce outputs that closely align with these reward models. However, human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses ( Casper et al. 2023 ) . This variability can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process. For example, feedback that favors supportive and enthusiastic responses might inadvertently lead the reward model to prioritize overly agreeable responses, which could result in sycophantic behavior ( Sharma et al. 2023 ; Perez et al. 2022 ) . This issue highlights the need for robust methods that precisely evaluate the impact of feedback on reward models, enabling humans to detect biased feedback and refine their feedback strategies more effectively. In this work, we assess the impact of human feedback on reward models by utilizing influence functions ( Hampel 1974 ; Koh and Liang 2017 ) . However, a significant challenge arises when applying influence functions to reward models, especially large-parameter models like LLMs and those involving extensive preference datasets, due to the high computational costs involved. To address this, we introduce a compute-efficient method that utilizes vector compression techniques ( Li and Li 2023 ) alongside the influence estimation method ( Kwon et al. 2024 ) , achieving a 2.5-fold speed acceleration compared to previous methods in computing influence functions. This approach significantly reduces the computational costs required to compute influence functions, facilitating more practical applications in large-scale settings. Figure 1: An overview of our work, which applies influence functions to reward modeling. We apply influence functions to critical tasks such as labeler bias detection and labeling strategy oversight, enhancing the interpretability of human feedback in RLHF. We demonstrate two applications of influence functions (see Figure 1 for an overview): (1) detecting labeler bias in training datasets, and (2) improving suboptimal labeling strategies. In our first experiment, we explore two prevalent biases in the RLHF paradigm: length and sycophancy bias, where labelers may naively prefer longer ( Saito et al. 2023 ) and more sycophantic responses ( Sharma et al. 2023 ) , regardless of response quality. To evaluate our approach, we introduce labeler bias by flipping a small portion of the Anthropic-HH dataset Bai et al. 2022 and assess whether they can be detected using influence functions. Our approach significantly outperforms several baselines, including GPT-4o ( OpenAI 2024 ) and various outlier detection methods ( Lee et al. 2018 ; Sun et al. 2022 ) , by effectively identifying biased samples. We further extend our analysis to original Anthropic-HH datasets (i.e., without flipping) using human surveys, confirming its robustness. Specifically, when analyzing 100 samples flagged by our approach, human annotators found 47 mislabeled cases, indicating the prevalence of labeler bias in real-world datasets. These results demonstrate the potential of our method to enhance publicly available preference datasets by accurately detecting and mitigating labeler biases. Additionally, we showcase the utility of influence functions in refining feedback strategies to better align with expert evaluations using a proof-of-concept experiment. Utilizing the Helpsteer 2 dataset ( Wang et al. 2024 ) , we simulate a scenario where an expert labeler, Alice, employs an optimal labeling strategy, and a non-expert labeler, Bob, uses a suboptimal one. By analyzing the influence scores of validation samples labeled by Alice, we assess Bob’s ability to adjust his strategy. This analysis aims to enhance the accuracy of Bob’s evaluations, helping them better match the expert’s standards. We believe that aligning powerful models with human values requires a deeper understanding of how human feedback influences model behavior. Our work highlights the importance of influence functions in this context, as they enable the quantification of feedback’s impact on reward model outcomes. Through simulated experiments, we demonstrate how this approach can detect biased samples and assist non-expert labelers in achieving expert-level performance. By enhancing the interpretability of human feedback in reward modeling, our approach can help labelers provide accurate feedback to reward models at complex tasks, contributing to scalable oversight ( Amodei et al. 2016 ; Bowman et al. 2022 ) . 2 Related Work Influence Functions Influence functions originate from robust statistics ( Hampel 1974 ) , which aim to robustly fit models based on the majority of the data, allowing accurate detection of outliers ( Tyler 2008 ; Rousseeuw and Leroy 2003 ) . Leveraging this property, influence functions have been applied to Deep Neural Networks to estimate the impact of data points on model behavior ( Koh and Liang 2017 ) . Specifically, it has been used on image datasets to detect adversarial samples or augmenting data ( Lee et al. 2020 ; Cohen et al. 2020 ) . It has also been applied to text classification tasks ( Schioppa et al. 2022 ; Guo et al. 2021 ) . Given their broad applicability, we extend the use of influence functions to reward modeling, specifically to measure the impact of human feedback on reward models. We build on recent advancements in efficient influence computation methods, which enable the estimation of influence functions even for large language models (LLMs) ( Kwon et al. 2024 ; Lin et al. 2024 ; Grosse et al. 2023 ) . Based on DataInf ( Kwon et al. 2024 ) , we apply influence functions to LLM-based reward models to analyze labeler bias and improve alignment in reward modeling. Scalable Oversight As AI models become more powerful, reliably providing feedback on their behavior becomes increasingly challenging ( Burns et al. 2024 ) . For instance,