Efficient Estimation of Kernel Surrogate Models for Task Attribution

Zhenshuo Zhang, Minxuan Duan *, Hongyang R. Zhang * · 2026

Read Efficient Estimation of Kernel Surrogate Models for Task Attribution by Zhenshuo Zhang, Minxuan Duan *, Hongyang R. Zhang * (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03783.txt.

Description

When one model trains on translation, code, mathematics, and text prediction together, which task taught it what? Zhenshuo Zhang and 2 co-authors study task attribution in a 2026 paper. They use surrogate models that predict target-task performance for any subset of training tasks. The exact alternative, retraining without each task in turn, is infeasible at scale.

Linear surrogates miss nonlinear interactions between tasks, such as XOR-type effects. The authors connect linear surrogates to influence functions through a second-order analysis, then introduce kernel surrogates that capture second-order interactions. A gradient-based estimator learns them efficiently from a first-order approximation of the pretrained model. Kernel surrogates track leave-one-out ground truth far better than baselines and improve downstream data selection substantially.

Catalog record

CREATORZhenshuo Zhang, Minxuan Duan *, Hongyang R. Zhang *
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03783

Browse the full library · About the Tower of Babel