LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution
Shuangqi Li, Hieu Le, Jingyi Xu, Mathieu Salzmann · 2026
Read LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution by Shuangqi Li, Hieu Le, Jingyi Xu, Mathieu Salzmann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21929.txt.
Introduction
Shuangqi Li, Hieu Le, Jingyi Xu, and Mathieu Salzmann ask which training examples shaped a model's answer. Training data attribution answers that question, and the best methods use gradients to define an influence function. The problem is scale: computing those gradients for large models is very expensive.
Popular shortcuts use random projection to shrink the gradient data. The team's 2026 preprint proposes LoRIF, which uses low-rank influence functions to keep attribution scalable. Knowing which examples drive a prediction helps teams find mislabeled data, trace strange outputs, and credit data sources. As models train on ever larger collections, attribution that cannot scale simply cannot be used.
Catalog record
| CREATOR | Shuangqi Li, Hieu Le, Jingyi Xu, Mathieu Salzmann |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21929
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution Shuangqi Li, Hieu Le, Jingyi Xu, Mathieu Salzmann Abstract Training data attribution (TDA) identifies which training examples most influenced a model’s prediction. The best-performing TDA methods exploits gradients to define an influence function. To overcome the scalability challenge arising from gradient computation, the most popular strategy is random projection (e.g., TRAK, LoGRA). However, this still faces two bottlenecks when scaling to large training sets and high-quality attribution: (i) storing and loading projected per-example gradients for all N N training examples, where query latency is dominated by I/O; and (ii) forming the D × D D\times D inverse Hessian approximation, which costs O ( D 2 ) O(D^{2}) memory. Both bottlenecks scale with the projection dimension D D , yet increasing D D is necessary for attribution quality—creating a quality–scalability tradeoff. We introduce LoRIF (Low-Rank Influence Functions) , which exploits low-rank structures of gradient to address both bottlenecks. First, we store rank- c c factors of the projected per-example gradients rather than full matrices, reducing storage and query-time I/O from O ( D ) O(D) to O ( c D ) O(c\sqrt{D}) per layer per sample. Second, we use truncated SVD with the Woodbury identity to approximate the Hessian term in an r r -dimensional subspace, reducing memory from O ( D 2 ) O(D^{2}) to O ( D r ) O(Dr) . On models from 0.1B to 70B parameters trained on datasets with millions of examples, LoRIF achieves up to 20 × \times storage reduction and query-time speedup compared to LoGRA, while matching or exceeding its attribution quality. LoRIF makes gradient-based TDA practical at frontier scale. Table 1 : Different families of TDA methods. The attribution quality of projection-based influence functions is heavily dependent on the projection dimension D , as shown in Figure 1 . Projection-based methods represent the current scalability frontier, and LoRIF pushes this frontier further. Method Family Scalability Accuracy Main bottleneck(s) Simulation-based methods Very poor Good Many counterfactual training runs Trajectory-aware methods Poor Good Computing through the training trajectory/checkpoints Infl. functions (parameter-space) Poor Fair Recomputating high-dimensional per-example gradients Infl. functions (projection-based) Fair Fair Storing and loading projected gradients; forming H^{-1} Grad./Repr. similarity Fair Poor Storing and loading/Limited attribution faithfulness LoRIF (ours) Good Fair — 1 Introduction Training data attribution (TDA) answers a fundamental question: which training examples were most responsible for a given model’s prediction on a test input? This capability enables diverse applications such as debugging incorrect predictions by tracing them to mislabeled or anomalous training data, detecting data poisoning attacks, curating high-quality training sets, and understanding how models generalize ( 20 ; 18 ) . As models grow larger and are trained on increasing web-scale datasets, the need for reliable, scalable attribution methods becomes more pressing. Influence functions ( 20 ) provide a theoretically-grounded approach to TDA: they approximate leave-one-out retraining via the use of gradients and the inverse Hessian. However, all existing methods for computing influence functions are expensive—iterative solvers require repeated Hessian–vector products across the training set, while simpler similarity methods sacrifice attribution accuracy for speed ( Section 2.2 ). Modern variants—TRAK ( 25 ) , LoGRA ( 8 ) , and TrackStar ( 5 ) —improve scalability by storing projected per-example gradients and using a damped Gauss–Newton Hessian approximation. However, while these projection-based methods represent the current scalability frontier for gradient-based TDA, they still face fundamental bottlenecks at scale. Specifically, the bottlenecks of projection-based TDA methods are twofold. First, gradient storage and I/O : storing projected per-example gradients for all N training examples requires O(ND) space, where D is the effective projection dimension (the total dimension of the stored gradient per example); at query time, loading these gradients dominates latency (around 96% of the query time). Second, inverse Hessian scaling : forming and storing the D\times D inverse Hessian approximation costs O(D^{2}) memory. Both bottlenecks scale with D —yet increasing D is key to high attribution quality. This creates a quality-scalability tradeoff: accurate attribution (large D ) is impractical, while tractable attribution (small D ) degrades quality. Here, we introduce LoRIF, which breaks this tradeoff by exploiting a key property: neural-network gradients have low effective rank (see Section 2.3 ), both within layers (per-example) and in the aggregate across examples. To address the storage and I/O bottleneck, we therefore store each projected per-example gradient as a rank- c factorization, reducing storage and query-time I/O from O(D) to O(c\sqrt{D}) per layer. To address the D\times D memory bottleneck, we compute a truncated SVD of the gradient matrix and exploit the Woodbury identity to compute the curvature term in an r -dimensional subspace without forming a dense D\times D inverse—reducing memory cost from O(D^{2}) to O(Dr) . Together, these approximations enable increasing D (improving quality) while keeping both storage/I/O and curvature costs tractable. We evaluate LoRIF on language models spanning three orders of magnitude: GPT2-small (124M parameters, 233k training examples), Olmo3-7B (7B parameters, 2.2M training examples), and Apertus-70B (70B parameters, 3.8M training examples). Compared to the state-of-the-art LoGRA, LoRIF yields • Storage: 2.3 \sim 20 times reduction, • Query-time speedup: 1.3 \sim 20 times faster influence computation, • Quality: matching or exceeding LDS and Tail-patch scores. In other words, LoRIF makes gradient-based TDA practical at frontier scale without sacrificing attribution quality. To our knowledge, this is the first time that gradient-based TDA is practical at this scale, without the need to recompute per-example gradients for each query batch. 2 Background and Related Work 2.1 Influence Functions A theoretically-grounded way to estimate the impact of a training sample on a model’s prediction consists of defining an influence function based on the model’s gradients. Below, we provide some background on this approach, before delving into the attempts that have been made to improve the scalability of this strategy. Classical influence functions. Given a model with parameters \theta^{*} trained to minimize an empirical risk, the influence of a training sample x_{tr} on a test sample x_{te} can be approximated as ( 20 ) \mathcal{I}(x_{tr},x_{te})=g_{te}^{\top}H^{-1}g_{tr}, (1) where g_{tr} and g_{te} are the gradients of the loss with respect to the model parameters on the training and test samples, and H is the Hessian of the training loss. This formulation approximates how the model’s prediction on the test sample x_{te} would change if the training sample x_{tr} were infinitesimally upweighted. Gauss-Newton Hessian approximation. For deep models, the Hessian matrix H is very large and typically positive semi- definite, making it intractable to compute and invert. As such, recent works have approximated the Hessian using the damped Gauss-Newton matrix ( 31 ; 4 ; 11 ; 8 ; 5 ) H\approx G^{\top}G+\lambda I, (2) where G\in\mathbb{R}^{N\times D} is the matrix of gradients (or projected gradients, as discussed below) for all N training samples, and \lambda>0 is a damping term. With this approximation, the influence function in Equation 1 becomes: \mathcal{I}(x_{tr},x_{te})\approx g_{te}^{\top}(G^{\top}G+\lambda I)^{-1}g_{tr}. (3) However, calculating Eq. ( 2 ) in the original parameter space is often intractable, as the gradient can have billions of dimensions. Inverse Hessian-vector product. Instead of explicitly forming the Hessian or its approximation, one can directly approximate the inverse Hessian-vector products (iHVP). Early iHVP approximations include stochastic Neumann-series estimators