Bayesian Influence Functions for Hessian-Free Data Attribution

Philipp Alexander Kreer, Wilson Wu, Maxwell Adam, Zach Furman, Jesse Hoogland · 2025

Read Bayesian Influence Functions for Hessian-Free Data Attribution by Philipp Alexander Kreer, Wilson Wu, Maxwell Adam, Zach Furman, Jesse Hoogland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26544.txt.

Introduction

Philipp Alexander Kreer and co-authors wrote this machine learning paper, published in two thousand twenty-five, on tracing a model's behavior back to its training data. The classic tool for this, influence functions, breaks down on deep networks because it needs a matrix inverse that often does not exist.

Their local Bayesian influence function swaps that inverse for statistics of the loss landscape, sampled with a stochastic-gradient Monte Carlo method. The swap captures higher-order links between the model's settings that the classic tool misses. It also scales to networks with billions of parameters. On tests that predict what retraining would do, the method reached state-of-the-art accuracy. The authors present it as a practical way to audit which training examples shaped a big model's answer.

Catalog record

CREATORPhilipp Alexander Kreer, Wilson Wu, Maxwell Adam, Zach Furman, Jesse Hoogland
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-26544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bayesian Influence Functions for Hessian-Free Data Attribution Philipp Alexander Kreer, Wilson Wu, Maxwell Adam, Zach Furman, Jesse Hoogland Abstract Classical influence functions face significant challenges when applied to deep neural networks, primarily due to non-invertible Hessians and high-dimensional parameter spaces. We propose the local Bayesian influence function (BIF), an extension of classical influence functions that replaces Hessian inversion with loss landscape statistics that can be estimated via stochastic-gradient MCMC sampling. This Hessian-free approach captures higher-order interactions among parameters and scales efficiently to neural networks with billions of parameters. We demonstrate state-of-the-art results on predicting retraining experiments. Figure 1: From influence functions (IF) to Bayesian influence functions (BIF): We introduce the local Bayesian Influence Function (BIF), which replaces the Hessian inversion of classical Influence Functions (IF) with a covariance estimation over the local loss landscape. This approach is sensitive to higher-order geometry and scales to models with billions of parameters. 1 Introduction Training data attribution (TDA) studies how training data shapes the behaviors of deep neural networks (DNNs)—a foundational question in AI interpretability and safetyA standard approach to TDA is influence functions (IF), which measure how models respond to infinitesimal perturbations in the training distribution ( Cook 1977 ; Cook & Weisberg 1982 ) . While elegant, influence functions rely on calculating the inverse Hessian and, therefore, break down for modern DNNs. Theoretically, DNNs have degenerate loss landscapes with non-invertible Hessians, which violate the conditions needed to define influence functions. Practically, for large models, the Hessian is intractable to compute directly. To mitigate these problems requires architecture-specific approximations that introduce structural biases ( Martens & Grosse 2015 ; Ghorbani et al. 2019 ; Agarwal et al. 2017 ; George et al. 2018 ) . We propose a principled, Hessian-free alternative grounded in Bayesian robustness. The key change is to replace Hessian inversion with covariance estimation over the local posterior ( Giordano et al. 2017 ; Giordano & Broderick 2024 ; Iba 2025 ) . This distributional approach naturally handles the degenerate loss landscapes of DNNs and bypasses the need to compute the Hessian directly. For non-singular models where the classical approach is valid, the BIF asymptotically reduces to the classical IF ( Appendix A ), which establishes the BIF as a natural generalization of the classical IF for modern deep learning. Contributions. We make the following contributions: • A theoretical extension of Bayesian influence functions to the local setting that enables the BIF to be applied to individual deep neural network checkpoints ( Section 2 ). • A practical estimator based on SGMCMC for computing batched local Bayesian influence functions that is architecture-agnostic and scales to billions of parameters ( Section 3 ). • Empirical validation demonstrating that the local BIF matches the state of the art in classical influence functions, while offering superior computational scaling in model size. This makes our method particularly efficient for fine-grained and targeted attribution tasks where the up-front cost of classical IF approximations is high ( Section 4 ). 2 Theory We first review classical influence functions ( Section 2.1 ), then review Bayesian influence functions ( Section 2.2 ), and finally propose our local extension ( Section 2.3 ). 2.1 Classical Influence Functions Classical influence functions quantify how a model would differ under an infinitesimal perturbation to its training data. Setup. We consider a training dataset \mathcal{D}_{\text{train}}=\{{\mathbf{z}}_{i}\}_{i=1}^{n} and a model parameterized by {\bm{w}}\in\mathcal{W}\subset\mathbb{R}^{d} . We define the empirical risk L_{\text{train}}({\bm{w}})=\sum_{i=1}^{n}\ell_{i}({\bm{w}}) , where \ell_{i}({\bm{w}})=\ell({\mathbf{z}}_{i};{\bm{w}}) is the loss for sample {\mathbf{z}}_{i} . We assume L_{\text{train}} is continuously second-differentiable and that our training procedure finds parameters {\bm{w}}^{*}\in\mathcal{W} at a local minimum, i.e., \nabla_{\bm{w}}L_{\text{train}}({\bm{w}}^{*})=0 . Influence on observables. We want to predict how the value of an observable \phi({\bm{w}})\colon\mathcal{W}\to\mathbb{R} would change under a perturbation to the training data. In particular, we’re interested in predicting the response of a query sample’s loss \phi({\bm{w}})=\ell({\mathbf{z}}_{j};{\bm{w}}) ) To model perturbation, we introduce importance weights \bm{\beta}=(\beta_{1},\ldots,\beta_{n}) and define the tempered risk L_{\text{train},\,\bm{\beta}}({\bm{w}})=\sum_{i=1}^{n}\beta_{i}\ell_{i}({\bm{w}}) . Assuming the loss Hessian is invertible, the implicit function theorem guarantees a neighborhood U_{{\bm{w}}^{*}}\ni{\bm{w}}^{*} such that, for all \bm{\beta} sufficiently close to \bm{1} , there is a unique minimizer of the tempered risk in this neighborhood {\bm{w}}^{*}(\bm{\beta})=\arg\min_{{\bm{w}}\in U_{{\bm{w}}^{*}}}L_{\text{train},\,\bm{\beta}}({\bm{w}}) . Note that {\bm{w}}^{*}(\bm{1})={\bm{w}}^{*} and that the function {\bm{w}}^{*}(-) depends on the starting {\bm{w}}^{*} ; in this sense, the classical influence is naturally local to a choice of parameters {\bm{w}}^{*} . The classical influence of training sample {\mathbf{z}}_{i} on the observable \phi evaluated at the optimum is defined as the sensitivity of \phi({\bm{w}}^{*}(\bm{\beta})) to the weight \beta_{i} : \CIF({\mathbf{z}}_{i},\phi):=\frac{\partial\phi({\bm{w}}^{*}({\bm{\beta}}))}{\partial\beta_{i}}\bigg\rvert_{{\bm{\beta}}=\bm{1}} (1) Applying the chain rule and the implicit function theorem, we arrive at the central formula: \boxed{\CIF({\mathbf{z}}_{i},\phi)=-\nabla_{\bm{w}}\phi({\bm{w}}^{*})^{\top}{\bm{H}}({{\bm{w}}^{*}})^{-1}\nabla_{{\bm{w}}}\ell_{i}({\bm{w}}^{*}),} (2) where {\bm{H}}({{\bm{w}}^{*}}) is the Hessian of L_{\text{train}} evaluated at {\bm{w}}^{*} . Figure 2: The per-token BIF captures semantic relationships in Pythia-2.8B. The posterior correlation (negative of the normalized BIF) between tokens is maximized for relationships like translations, alternate spellings, and synonyms. 2.2 Bayesian Influence Functions An alternative perspective, grounded in Bayesian learning theory and statistical physics, avoids the Hessian by considering a distribution over parameters instead of a single point estimate. Influence on observable expectations. We obtain the Bayesian influence \BIF(z_{i},\phi) of sample {\mathbf{z}}_{i} on an observable \phi by replacing the point estimate \phi({\bm{w}}^{*}) in Equation 1 with an expectation value \mathbb{E}_{\text{train},\,\bm{\beta}}[\phi({\bm{w}})] : \BIF({\mathbf{z}}_{i},\phi):=\frac{\partial\mathbb{E}_{\text{train},\,\bm{\beta}}[\phi({\bm{w}})]}{\partial\beta_{i}}\bigg\rvert_{{\bm{\beta}}=\bm{1}}. (3) Here, \mathbb{E}_{\text{train},\,\bm{\beta}}[\phi({\bm{w}})]=\int\phi({\bm{w}})p_{{\bm{\beta}}}({\bm{w}}\mid\mathcal{D}_{\text{train}})\,d{\bm{w}} is an expectation over a tempered Gibbs measure p_{{\bm{\beta}}}({\bm{w}}\mid\mathcal{D}_{\text{train}})\propto\exp(-L_{\text{train},\,\bm{\beta}}({\bm{w}}))\varphi({\bm{w}}) with prior \varphi({\bm{w}}) . This is a tempered Bayesian posterior if the loss is a negative log-likelihood \ell_{i}({\bm{w}})=-\log p({\mathbf{z}}_{i}\mid{\bm{w}}) , which we assume is the case for the rest of the paper. A standard result from statistical physics (see Baker et al. 2025 ) relates the derivative of the expectation to a covariance over the untempered ( \bm{\beta}=\bm{1} ) posterior under mild regularity conditions: \boxed{\BIF({\mathbf{z}}_{i},\phi)=-\mathrm{Cov}(\ell_{i}({\bm{w}}),\phi({\bm{w}})).} (4) Bayesian influence is the negative covariance between an observable and the sample’s loss over the tempered posterior. In Section A.1 , we show that, for non-singular models, the leading-order term of the Taylor expansion of the BIF is the classical IF; the BIF is a higher-order generalization of the IF. 2.3 Local Bayesian Influence Functions Computing expectations over the global Bayesian posterior p({\bm{w}}\mid\mathcal{D}_{\text{train}}) is generally intractable for DNNs. Furthermore, standard DNN training yields individual checkpoints {\bm{w}}^{*} , and we are often most interested in the influence local to this final trained model. Therefore, we adapt the BIF with a localization mechanism. Following Lau et al. 2025 , we define a localized Bayesian posterior by replacing the prior \varphi({\bm{w}}) with an isotropic Gaussian with precision \gamma centered at the parameters {\bm{w}}^{*} : p_{\gamma}({\bm{w}}\mid\mathcal{D}_{\text{train}},{\bm{w}}^{*})\propto\exp\left\lparen-\sum_{i=1}^{n}\ell_{i}({\bm{w}})-\frac{\gamma}{2}\lVert{\bm{w}}-{\bm{w}}^{*}\rVert_{2}^{2}\right\rparen. (5) The local Bayesian influence function (local BIF) is defined as in Equation 4 but via a covariance over the localized Gibbs measure, indicated by the index \gamma : \boxed{\BIF_{\gamma}({\mathbf{z}}_{i},\phi)=-\mathrm{Cov}_{\gamma}(\ell_{i}({\bm{w}}),\phi({\bm{w}})).} (6) For comparison, note that classical IFs are ill-defined for singular models, such as neural networks with non-invertible Hessians. A common practical remedy is to use a dampened Hessian ({\bm{H}}({{\bm{w}}^{*}})+\gamma{\bm{I}}) . This is equivalent to adding an \ell_{2} regularizer centered at {\bm{w}}^{*} to the loss, which is the same trick we use in defining \BIF_{\gamma} . In Section A.2 , we show that the first-order term of the expansion of the local BIF is the dampened IF (with a vanishing dampening factor); the local BIF is a higher-order generalization of the

Read the complete text