Layer-Aware Influence for Online Data Valuation Estimation

Ziao Yang, Longbo Huang, Hongfu Liu · 2025

Read Layer-Aware Influence for Online Data Valuation Estimation by Ziao Yang, Longbo Huang, Hongfu Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16007.txt.

Introduction

Ziao Yang, Longbo Huang, and Hongfu Liu estimate the value of each training sample while training is still running. Data-centred learning improves models by curating good samples rather than by redesigning networks, so sample influence must be scored. Past work scores influence once, on a finished model, and misses how a sample's worth shifts during optimisation.

Their layer-aware online estimator needs only gradients from loss to output, avoiding both per-parameter and full-network gradients. That design keeps the ranking of samples faithful while slashing the cost of repeated scoring. Across large language model pretraining, fine-tuning, and image classification, the method improves accuracy at much lower time and memory cost than earlier estimators. Dynamic curation thus becomes practical at scale, the three authors argue.

Catalog record

CREATORZiao Yang, Longbo Huang, Hongfu Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16007

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Layer-Aware Influence for Online Data Valuation Estimation Ziao Yang, Longbo Huang, Hongfu Liu Abstract Data-centric learning emphasizes curating high-quality training samples to boost performance rather than designing new architectures. A central problem is to estimate the influence of training sample efficiently. Prior studies largely focus on static influence measured on a converged model, overlooking how data valuation dynamically changes during optimization. This omission neglects the dynamic nature of sample influence during optimization, especially in deep models. To address the computational burden of frequent influence estimation, we develop a layer-aware online estimator that requires only loss-to-output gradients. This design avoids parameter-level and full-network gradients while preserving ranking fidelity. Extensive experiments across LLM pretraining, fine-tuning, and image classification show our method improves accuracy with substantially lower time and memory cost, making dynamic data curation efficient and scalable in practice. 1 Introduction Data-centric learning surges as an emerging topic in machine learning community, which focuses on curating high-quality training samples for performance boosting, rather than designing novel algorithms [ 1 , 2 , 3 ] . Sample influence estimation, also known as data attribution or data valuation, is the fundamental research question in data-centric learning, which assesses the sample importance associated with a certain model. Based on that, detrimental samples can be identified, which will be removed from the training set for another round training with the expected performance gain. Sample influence estimation can generally be categorized into two categories [ 4 ] . 1) Retraining-based methods include the classical leave-one-out influence approach [ 5 ] that retrains models and observes performance changes after removing one training sample. While useful as an ideal baseline, this is computationally untenable on deep models. Other representative methods such as model-agnostic Shapley value approaches [ 6 , 7 , 8 ] also suffer similar problems. Computationally efficient approaches, such as KNN-Shap [ 9 ] , are limited to using KNN classifiers and their recursive calculations. 2) Gradient-based methods can be used to approximately estimate influence without expensive overheads of retraining. Influence functions [ 1 , 10 , 11 ] are representative tools in this category, which employ the first-order Taylor expansion to estimate the performance change with sample gradients. Although model retraining is avoided, the inverse of the Hessian matrix in influence functions brings great challenges for large deep models. Several pioneering studies focus on approximating the inverse of the Hessian matrix in an efficient way [ 1 , 12 , 13 ] . While the above pioneering studies shed light on model-associated sample influence for identifying beneficial or detrimental samples in terms of data quality, they overlook another crucial factor—online data valuation dynamically change during optimization, particularly in deep models. This introduces a critical limitation: samples identified as detrimental based on the initial model may no longer be detrimental for the newly trained model after their removal. Such oversight not only necessitates two rounds of training but also results in suboptimal performance and fails to harness the full potential of online data valuation-aware training. Contributions . To address the aforementioned limitations, we introduce a layer-aware online data valuation framework that estimates per-sample influence during training and integrates naturally with SGD-style updates. Our estimator aligns scoring with optimization and can be instantiated as policies such as online curation, reweighting, or priority sampling—without a filter–then–retrain cycle. Our key contributions are summurized as follows: • Research Question: We formalize online data valuation that jointly accounts for data quality during optimization, addressing the scoring–optimization mismatch and order-dependent interactions that static methods ignore. • Technical Innovation: We design a Hessian-free, layer-aware online influence estimator that backpropagates only to the model outputs, avoiding full-network parameter gradients. Our lightweight calibration mitigates cross-layer scale bias while preserving valuation fidelity, enabling single-run training with minimal overhead and seamless integration with SGD. • Experimental Validation: Extensive experiments are conducted across diverse scenarios, including LLM pre-training, LLM fine-tuning, and both image and text classification. The results demonstrate that our proposed method is highly effective and also computationally efficient. 2 Related Work In this section, we introduce the data valuation with a focus on influence function, and other online data curation topics including curriculum learning and data scheduling. Influence function . Influence functions, originally developed in robust statistics [ 14 , 5 , 15 ] , quantify the sensitivity of model parameters to perturbations in training data. Introduced to the machine learning community by Koh and Liang 2017 , they have since been widely applied to tasks such as identifying detrimental sample, detecting outliers, and spotting mislabeled samples. Using a first-order Taylor expansion, influence functions estimate sample influence based on the model’s performance on a validation set, leveraging the inverse of the Hessian matrix and sample gradients—eliminating the need for model retraining. However, computing the inverse of the Hessian matrix is computationally prohibitive, particularly for large deep models. Several pioneering studies address this challenge by approximating the inverse of Hessian matrix more efficiently. LiSSA [ 1 ] employs Hessian-vector products for approximation, while EKFAC [ 12 ] incorporates efficient eigen decomposition techniques. DataInf [ 13 ] further simplifies influence calculations for large models by substituting the Hessian inverse with a rank-1 closed-form expression. Beyond the static estimation of sample influence based on a final model (sometimes referred to as a checkpoint), recent approaches focus on dynamic influence estimation. GEX [ 16 ] leverages geometric ensembles of multiple checkpoints to approximate influence functions, mitigating bilinear constraints and addressing non-linear losses. Similarly, TDA [ 17 ] introduces a checkpoint-based segmentation strategy that combines implicit differentiation and unrolling, leveraging EKFAC [ 12 ] for efficiency. TRAK [ 18 ] employs randomly-projected kernel to reduce the dimensional of the Hessian matrix for tractable computing. Recently, a surge of interest in Hessian-free influence functions has emerged, which replaces the Hessian inverse with a simple identity matrix [ 19 , 20 , 11 , 21 ] . These approaches offer computational simplicity and scalability, making them particularly attractive for deep learning models with competitive performance. While the above studies have made significant advancements in data-centric learning, they largely overlook the dynamic variance of sample influence throughout the model training process. Although some recent works attempt to address dynamic influence estimation, their approach—simply summing up sample influences at different checkpoints—fails to capture the evolving nature of sample influence effectively. This limitation arises because model parameters at early checkpoints often differ substantially from those at later stages, rendering static aggregation methods inadequate. Furthermore, these studies neglect the role of online data valuation change during the optimization, which is particularly critical for deep models. By treating data valuation as static, existing methods miss opportunities to dynamically adapt training strategies based on evolving sample importance, ultimately compromising both efficiency and effectiveness in influence-aware learning. Curriculum learning . Curriculum learning [ 22 , 23 , 24 ] is a training strategy that incorporates data sequence into model optimization by presenting data in an easy-to-hard order, mimicking the structured learning process observed in human education. As a general training paradigm, curriculum learning has been widely applied across diverse domains, including supervised learning tasks in computer vision [ 25 , 26 ] and natural language processing [ 27 , 28 ] , as well as reinforcement learning [ 29 , 30

Read the complete text