Influence Functions for Preference Dataset Pruning
Daniel Fein & Gabriela Aranguiz-Dias · 2025
Read Influence Functions for Preference Dataset Pruning by Daniel Fein & Gabriela Aranguiz-Dias (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14344.txt.
Introduction
Daniel Fein and Gabriela Aranguiz-Dias tested influence functions in twenty twenty-five for pruning noisy human preference data. Preference data sets steer language model fine-tuning but contain harmful examples mixed with good ones. Influence functions estimate how much each training example moves performance on a validation set, if approximated cheaply enough to use.
Adapting a summarization data set for reward model training, they filtered examples with conjugate-gradient influence estimates. Removing the worst ten percent and retraining lifted accuracy by about one and a half points. A surprise: plain gradient similarity found helpful examples better than influence functions did. The authors read this as evidence that local curvature matters for spotting harmful examples, less so for finding helpful ones.
Catalog record
| CREATOR | Daniel Fein & Gabriela Aranguiz-Dias |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-14344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Influence Functions for Preference Dataset Pruning Daniel Fein & Gabriela Aranguiz-Dias Abstract Language models are commonly fine-tuned via reinforcement learning to alter their behavior or elicit new capabilities. Datasets used for these purposes, and particularly human preference datasets, are often noisy. The relatively small size post-training datasets, combined with parameter-efficient fine-tuning methods, enable the use of influence functions approximations to detect and prune training examples that are harmful to performance on a validation set. In this work, we adapt the TL;DR dataset for reward model training to demonstrate how conjugate-gradient approximated influence functions can be used to filter datasets. In our experiments, influence function filtering yields a small retraining accuracy uplift of 1.5% after removing 10% of training examples. We also show that gradient similarity outperforms influence functions for detecting helpful training examples. This suggests that local curvature is important for detecting harmful training examples, but less so for identifying helpful examples. 1 Introduction Influence functions are powerful tools for understanding how specific data affect the behavior of neural networks trained via gradient descent ( Koh & Liang 2017 ) . However, their practicality for language models is encumbered by their reliance on the inverse-hessian with respect to model parameters. The large scale of language models, both in terms of parameter count and training data, thus poses a challenge for influence-function based data attribution on the scale of pre-training. Despite this, approximations of the inverse hessian used at scale have demonstrated the power of influence functions to reveal insights into how language models make use of their training data Grosse et al. 2023 . Recently, the post-training paradigm has proven crucial for controlling the behavior of language models both with respect to helpfulness and harmlessness as well as math and coding abilities ( Bai et al. 2022b ; Zelikman et al. 2022 ) . Meanwhile, techniques for parameter-efficient fine-tuning such as Low-Rank Adaptation enable this post-training to be done via training of relatively few model parameters Hu et al. 2022 . The outsized effectiveness of this much less computationally-expensive training warrants a new investigation into influence functions for understanding the relationship between data and model behavior. Turning specifically to the task of human preference modeling, it is estimated that 20-40% of training data used to align language models with human preferences is noisy ( Gao et al. 2024 ) . Influence functions have been used to reduce the size of training sets for computer vision tasks ( Yang et al. 2023 ) , but have not yet been applied to language model data curation. In this work, we demonstrate for the first time how influence functions approximated via the conjugate gradient method can help create stronger reward models via filtering out of noisy examples. 2 Related Works Influence Function Approximation Martens et al. 2010 first approximated the hessian of deep networks using the conjugate gradient method,and Koh & Liang 2017 applied this to influence functions. Kwon et al. 2023 and Grosse et al. 2023 propose algorithms that efficiently approximate the conjugate gradient method, and apply them to LoRA-finetuned and pretrained language models, respectively. They do not evaluate the conjugate gradient method for data curation, nor do they consider the task of preference modeling. RLHF and Data Curation Stiennon et al. 2020 showed that a language model could be taught to generate better summaries using human feedback in the form of pairwise preferences. We use the TL;DR dataset they provide. Bai et al. 2022a post-trained a language model to align it’s behavior to be helpful and harmless. Morimura et al. 2024 and Gao et al. 2024 point out and attempt to address low data quality for preference data. The former does this by proposing a framework for data collection, the latter by proposing filtering based on trained reward-model logits. 3 Methods 3.1 Task and Data We study the human–preference reward–modeling task introduced by Bai et al. 2022b on the public TL;DR dataset released by OpenAI. 1 1 1 https://github.com/openai/summarize-from-feedback Each training example is a chosen and a rejected summary of the same news article, labeled by crowd workers. Following common practice, we cast the task as a Bradley–Terry pairwise–ranking problem: the reward model f_{\theta} should assign a higher scalar reward to the chosen summary than to its rejected counterpart. Dataset Modification . In order to make the problem tractable under a very tight computation budget, we filter out all examples for which both summaries are longer than 24 tokens Llama3 tokens, and discard the original post text, using only summary preferences and summaries themselves. This leaves us with roughly 8.5k examples total. For influence estimation we sample |\mathcal{D}_{\text{val}}|=100 validation pairs uniformly at random from the held‑out validation set. All remaining validation data serve as an untouched test set for final evaluation. 3.2 Base Model and Fine‑tuning Setup Our base model is LLaMA-3.2‑1B . We apply Low‑Rank Adaptation ( LoRA ; Hu et al. 2022 ) with rank r{=}8 , \alpha{=}16 and dropout 0.05 to the projection matrices \{q,k,v,o\} . Fine‑tuning uses the AdamW optimiser ( \beta_{1}{=}0.9,\ \beta_{2}{=}0.98,\ \epsilon{=}10^{-8} ) for three epochs, batch size 124 and learning rate 1\times 10^{-5} with cosine decay. Only LoRA parameters ( \approx 0.12\% of total weights) are updated. 3.3 Influence–Function Approximation The goal is to estimate, for every training example z_{i} and validation example z_{j} , the classical influence value \mathcal{I}_{\text{IF}}(z_{i},z_{j})=-\nabla_{\theta}L(z_{j})^{\top}H_{\theta}^{-1}\nabla_{\theta}L(z_{i}) , where H_{\theta} is the Hessian of the total training loss. Below we give an intuitive walk‑through of the procedure, then collect the steps into Algorithm 1 . Instead of inverting a billion‑dimensional Hessian, we study only the LoRA adapter weights . These account for 0.12\% of all parameters ( \approx\!1.2 M scalars) yet dominate post‑training behavior ( Hu et al. 2022 ) . All gradients are therefore taken with respect to this small sub‑space, making both memory use and matrix‑vector multiplies tractable. The Optimization Perspective Computing H_{\theta}^{-1}\nabla_{\theta}L(z_{j}) is equivalent to solving the linear system (H_{\theta}+\lambda I)\,x=g_{j}\quad\text{with}\quad g_{j}:=\nabla_{\theta}L(z_{j}), where we add Tikhonov damping \lambda{=}10^{-2} to guarantee positive definiteness. This linear system is the first‑order optimality condition of the strictly convex quadratic \min_{x\in\mathbb{R}^{d}}\;\frac{1}{2}\,x^{\top}(H_{\theta}+\lambda I)x\;-\;g_{j}^{\top}x, so the problem can be viewed as an optimization problem whose solution x^{\star} is the desired inverse‑Hessian–vector product. To computing an HVP without forming H_{\theta} , we use the double‑back‑prop trick ( Pearlmutter 1994 ) . We average the operation over a mini‑batch of |\mathcal{B}|=20 training examples, reducing variance while staying limiting GPU memory use. Algorithm 1 Influence computation in the LoRA parameter space 1: LoRA‑tuned model f_{\theta} ; training set \mathcal{D}_{\text{train}} ; validation set \mathcal{D}_{\text{val}} ; damping \lambda ; batch size B ; CG iteration budget K . 2: for all z_{j}\in\mathcal{D}_{\text{val}} do \triangleright loop over validation pairs 3: g_{j}\leftarrow\nabla_{\theta}L(z_{j}) \triangleright retain comp. graph 4: x_{j}\leftarrow 0 ; r\leftarrow g_{j} ; p\leftarrow r 5: for k=1 to K do \triangleright conjugate‑gradient iterations 6: \mathcal{B}\leftarrow sample B elements from \mathcal{D}_{\text{train}} 7: h\leftarrow\dfrac{1}{\lvert\mathcal{B}\rvert}\sum_{z_{i}\in\mathcal{B}}\nabla_{\theta}^{2}L(z_{i})\,p\;+\;\lambda p \triangleright batched HVP via double back‑prop 8: \alpha\leftarrow\dfrac{\langle r,r\rangle}{\langle p,h\rangle} 9: x_{j}\leftarrow x_{j}+\alpha p 10: r_{\text{new}}\leftarrow r-\alpha h 11: if \lVert r_{\text{new}}\rVert<\varepsilon then 12: break 13: end if 14: \beta\leftarrow\dfrac{\langle r_{\text{new}},\,r_{\text{new}}\rangle}{\langle r,\,r\rangle} 15: p\leftarrow r_{\text{new}}+\beta p 16: r\leftarrow r_{\text{new}} 17: end for 18: x_{j}\leftarrow\textbf{detach}(x_{j}) \triangleright x_{j}\approx H_{\theta}^{-1}g_{j} 19: for all z_{i}\in\textbf{local\_shard}(\mathcal{D}_{\text{train}}) do 20: g_{i}\leftarrow\nabla_{\theta}L(z_{i}) 21: \mathcal{I}(z_{i},z_{j})\leftarrow-\langle x_{j},\,g_{i}\rangle \triangleright