Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining

Weiyi Wang, Junwei Deng, Yuzheng Hu, Shiyuan Zhang, Xirui Jiang, Runting Zhang, Han Zhao, Jiaqi W. Ma · 2025

Read Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining by Weiyi Wang, Junwei Deng, Yuzheng Hu, Shiyuan Zhang, Xirui Jiang, Runting Zhang, Han Zhao, Jiaqi W. Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24261.txt.

Introduction

Weiyi Wang and co-authors study a hidden cost in data attribution in a 2025 machine learning paper. Attribution methods score how much each training example shapes a model's behavior, a tool that data teams lean on more each year.

The team ran the first large-scale test of how touchy these methods are to their settings. Most methods did prove sensitive to a few key settings, sometimes strongly. Unlike normal models, they cannot be tuned cheaply, since scoring a setting often means retraining on data subsets. The authors argue that better theory about how the settings act is the practical way out, so users can pick values without burning huge compute.

Catalog record

CREATORWeiyi Wang, Junwei Deng, Yuzheng Hu, Shiyuan Zhang, Xirui Jiang, Runting Zhang, Han Zhao, Jiaqi W. Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24261

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining Weiyi Wang, Junwei Deng, Yuzheng Hu, Shiyuan Zhang, Xirui Jiang, Runting Zhang, Han Zhao, Jiaqi W. Ma Abstract Data attribution methods, which quantify the influence of individual training data points on a machine learning model, have gained increasing popularity in data-centric applications in modern AI. Despite a recent surge of new methods developed in this space, the impact of hyperparameter tuning in these methods remains under-explored. In this work, we present the first large-scale empirical study to understand the hyperparameter sensitivity of common data attribution methods. Our results show that most methods are indeed sensitive to certain key hyperparameters. However, unlike typical machine learning algorithms—whose hyperparameters can be tuned using computationally-cheap validation metrics—evaluating data attribution performance often requires retraining models on subsets of training data, making such metrics prohibitively costly for hyperparameter tuning. This poses a critical open challenge for the practical application of data attribution methods. To address this challenge, we advocate for better theoretical understandings of hyperparameter behavior to inform efficient tuning strategies. As a case study, we provide a theoretical analysis of the regularization term that is critical in many variants of influence function methods. Building on this analysis, we propose a lightweight procedure for selecting the regularization value without model retraining, and validate its effectiveness across a range of standard data attribution benchmarks. Overall, our study identifies a fundamental yet overlooked challenge in the practical application of data attribution, and highlights the importance of careful discussion on hyperparameter selection in future method development. 1 Introduction Data attribution methods quantify the contribution of individual training samples to a machine learning model’s predictions or overall performance [ 21 ] . These methods have been successfully applied to a range of data-centric tasks in modern AI, including training data selection [ 36 ] , mislabel detection [ 21 ] , and copyright compensation [ 7 ] . Recently, this field has witnessed a surge of novel methods [ 10 , 17 , 29 , 11 , 28 , 6 , 3 , 34 ] , each promising improvements in efficiency, efficacy, or theoretical soundness. Despite this rapid progress, a crucial practical aspect remains largely underexplored: the hyperparameter sensitivity of data attribution methods. Existing benchmark studies [ 18 , 8 ] largely adopt the hyperparameter settings used in the original papers proposing the methods. While some of the original papers include limited ablation studies, few of them offer practical guidance for hyperparameter selection, and many methods include implicit tunable knobs that have received little scrutiny. Moreover, the optimal hyperparameter choices can vary across datasets and models. Hyperparameter tuning is particularly challenging in data attribution due to the high computation costs associated with common evaluation metrics. Standard evaluation metrics—such as Leave - One - Out (LOO) correlation [ 21 ] and Linear Datamodeling Score (LDS) [ 28 ] —require retraining models on subsets of the full training dataset for hundreds or even thousands of times. A conventional hyperparameter search under such evaluation metrics is prohibitively expensive in practice. In this work, we present the first systematic, large-scale empirical study of hyperparameter sensitivity in data attribution. We benchmark a suite of popular data attribution methods across various datasets and models with extensive hyperparameter sweeps. Our results confirm that most methods are sensitive to certain key hyperparameters, and the optimal hyperparameter choices vary across experimental settings. Our analysis also reveals other interesting novel findings regarding the interaction of different hyperparameters. Our findings highlight hyperparameter tuning as a critical open challenge in the practical application of data attribution. As a first step towards addressing this challenge, we conduct a case study on selecting the regularization term—a crucial hyperparamter in many influence function methods [ 21 , 11 ] —without costly retraining. We first provide a theoretical analysis of how the regularization term affects the influence function performance in terms of LDS. Building on this analysis, we introduce a surrogate indicator that enables selection of the regularization value without model retraining. We validate the effectiveness of the proposed approach across multiple experimental settings, including several variants of influence functions on different datasets and models. To summarize, our contributions are as follows: • We identify hyperparameter tuning as a critical but largely overlooked challenge in the practical application of data attribution methods. • We conduct the first comprehensive empirical study of hyperparameter sensitivity in data attribution, benchmarking a range of widely used methods across diverse settings, confirming the necessity of practical hyperparameter selection strategies. • We provide a theoretical analysis of the regularization term in influence functions and propose an efficient, retraining-free selection procedure with strong empirical performance. These results advance both the understanding and practical deployment of data attribution methods, encouraging a more systematic treatment of hyperparameters as integral components in future methodological development. 2 Background In this section, we provide the necessary background to contextualize our empirical study of hyperparameter sensitivity in Section 3 and the subsequent case analysis in Section 4 . 2.1 Data Attribution Methods and Their Hyperparameters In this work, we focus on the methods included in a recent benchmark study [ 8 ] , which consists of state-of-the-art efficient data attribution methods while omitting a family of game-theoretic methods such as Data Shapley [ 10 , 17 ] . 1 1 1 While these game-theoretic methods are theoretically principled, they are typically computationally expensive and cannot scale up to even moderately large models. In the following, we introduce the notation, then review a set of most widely-used data attribution methods as well as their major hyperparameters examined in our study. Notation. Consider a multi-class classification problem with dataset S=\{(x_{i},y_{i})\}_{i=1}^{n}\subset\mathcal{Z} , where \mathcal{Z}=\mathcal{X}\times\mathcal{Y} , inputs x_{i}\in\mathcal{X}\subset\mathbb{R}^{d} , and labels y_{i}\in\mathcal{Y} . We use a parameterized hypothesis function h(\cdot,\theta)\mathrel{\mathop{\ordinarycolon}}\mathcal{X}\rightarrow\Delta_{|\mathcal{Y}|-1} with parameters \theta\in\mathbb{R}^{p} , which typically outputs class probabilities via a softmax layer. The cross-entropy loss for a single example z=(x,y) is denoted by L(z,\theta)=\ell(h(x,\theta),y) . The empirical risk at \theta over a subset A\subset S of size a is R_{A}(\theta)\mathrel{\mathop{\ordinarycolon}}=\frac{1}{a}\sum_{z\in A}L(z,\theta) . For a given test example z^{\prime}=(x^{\prime},y^{\prime}) , following Park et al. 2023 , we define the model output function as f(z^{\prime},\theta)\mathrel{\mathop{\ordinarycolon}}=\ln\frac{p(z^{\prime},\theta)}{1-p(z^{\prime},\theta)} , where p(z^{\prime},\theta)=\exp(-L(z^{\prime},\theta)) is the model’s predicted probability for the correct class y^{\prime} . We denote the parameter that minimizes R_{S} as \theta_{S}^{*} . Data attribution methods. A data attribution method (or an attributor ) is a function \tau\mathrel{\mathop{\ordinarycolon}}\mathcal{Z}\times S\rightarrow\mathbb{R} that, for a given test example z^{\prime}\in\mathcal{Z} , assigns \tau(z^{\prime},z_{i}) to each training point z_{i}\in S . A popular attributor is the Influence Function (IF) attributor, introduced by Koh and Liang 2017 , which computes the influence of a training example z_{i}\in S on a test example z^{\prime}\in\mathcal{Z} as \tau_{\mathrm{IF}}(z^{\prime},z_{i})\mathrel{\mathop{\ordinarycolon}}=-\nabla_{\theta}f(z^{\prime},\theta_{S}^{*})^{\top}H_{S}^{-1}\nabla_{\theta}L(z_{i},\theta_{S}^{*}), (1) where H_{S}\mathrel{\mathop{\ordinarycolon}}=\nabla_{\theta}^{2}R_{S}(\theta_{S}^{*}) is the Hessian of the empirical risk R_{S} . To bypass the prohibitive computation of H_{S} , they also propose using efficient approaches to approximate H_{S}^{-1}\nabla_{\theta}L(z_{i},\theta_{S}^{*}) , including conjugate gradient (CG) [ 26 ] and stochastic estimation method LiSSA [ 1 ] . For non-convex models, H_{S} is usually replaced with its regularized version H_{S}+\lambda I_{p} in case of singularity [ 21 ] , where \lambda>0 is a regularization term and I_{p} is the identity matrix of size p . To further handle non-convexity and

Read the complete text