GraSS : Scalable Data Attribution with Gradient Sparsification and Sparse Projection

Pingbang Hu, Joseph Melkonian, Weijing Tang, Han Zhao, Jiaqi W. Ma · 2025

Read GraSS : Scalable Data Attribution with Gradient Sparsification and Sparse Projection by Pingbang Hu, Joseph Melkonian, Weijing Tang, Han Zhao, Jiaqi W. Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18976.txt.

Description

Knowing which training examples shaped a model's answer is called data attribution, and the honest methods are expensive. Pingbang Hu and co-authors speed them up in a 2025 paper with a compression trick named GraSS. Gradient-based attribution needs a gradient for every training sample, which devours memory and time on large models.

GraSS exploits the fact that those per-sample gradients are mostly zeros. Sparsifying them and projecting them into a smaller space cuts the cost below linear growth. A variant tuned for plain linear layers, FactGraSS, ran up to 165 percent faster than the best prior methods on billion-parameter models. Accuracy of the influence estimates held up across experiments. The authors release their code openly for other teams to use.

Catalog record

CREATORPingbang Hu, Joseph Melkonian, Weijing Tang, Han Zhao, Jiaqi W. Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18976

Browse the full library · About the Tower of Babel