ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation

Yanzhou Pan, Huawei Lin, Yide Ran, Jiamin Chen, Xiaodong Yu, Weijie Zhao, Denghui Zhang, Zhaozhuo Xu · 2025

Read ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation by Yanzhou Pan, Huawei Lin, Yide Ran, Jiamin Chen, Xiaodong Yu, Weijie Zhao, Denghui Zhang, Zhaozhuo Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01052.txt.

Introduction

Yanzhou Pan and seven co-authors present ALinFiK, a method for pricing single training examples for large language models, in this 2025 arXiv paper. Good data drives model quality, and buyers with limited budgets need to know which samples earn their price. The authors define a score, the linearized future influence kernel, that estimates how much one sample will improve a model during training.

Computing that score directly is heavy, so ALinFiK learns to approximate it, making valuation fast enough for large collections. The pitch is a third-party service fair to both data sellers and model builders. In the team's tests, the method beat existing baselines on both accuracy of the ranking and speed, and its edge grew as the models got bigger.

Catalog record

CREATORYanzhou Pan, Huawei Lin, Yide Ran, Jiamin Chen, Xiaodong Yu, Weijie Zhao, Denghui Zhang, Zhaozhuo Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation Yanzhou Pan, Huawei Lin, Yide Ran, Jiamin Chen, Xiaodong Yu, Weijie Zhao, Denghui Zhang, Zhaozhuo Xu Abstract Large Language Models (LLMs) heavily rely on high-quality training data, making data valuation crucial for optimizing model performance, especially when working within a limited budget. In this work, we aim to offer a third-party data valuation approach that benefits both data providers and model developers. We introduce a linearized future influence kernel (LinFiK), which assesses the value of individual data samples in improving LLM performance during training. We further propose ALinFiK, a learning strategy to approximate LinFiK, enabling scalable data valuation. Our comprehensive evaluations demonstrate that this approach surpasses existing baselines in effectiveness and efficiency, demonstrating significant scalability advantages as LLM parameters increase. 1 Introduction Large Language Models (LLMs) Brown et al. (2020) ; Wei et al. (2022) ; Touvron et al. (2023) are transforming fields such as natural language processing Nazir et al. (2024) ; Si et al. (2024) ; Du et al. (2024) and automated reasoning Hao et al. (2024) ; Wen et al. (2024) ; Yang et al. (2024b) . Their effectiveness is closely tied to the quality of the training data. In real-world scenarios, model owners and data providers often have distinct objectives. Model owners aim to maximize model performance while minimizing costs, including data acquisition costs and training costs Sachdeva et al. (2024) . Conversely, data providers seek fair compensation for their contributions Pei (2020) . Third-Party Data Valuation: Our research presents a third-party data valuation algorithm that serves the needs of both model owners and data providers. By identifying high-value data subsets from the data provider and delivers them to model owners for training Sachdeva et al. (2024) ; Choe et al. (2024) . This process helps reduce data acquisition and training costs for model owners while ensuring data providers receive compensation based on the value their data contributes. For model owners, the platform efficiently selects the most impactful data from large datasets, minimizing acquisition and computational costs. Model owners can achieve improved performance with fewer resources by focusing on high-value data Biderman et al. (2023) . For data providers, the platform offers a transparent and quantitative method for valuing data, allowing them to monetize their contributions based on the proven impact on model performance Liang et al. (2018) . This promotes a fairer data marketplace Pei (2020) and encourages the creation of high-quality, relevant datasets. Challenges in Third-Party Data Valuation: We identify three major challenges in third-party data valuation for LLM training. (1) Future influence estimation: Because the goal is to perform pre-training data selection, our approach requires third-party platforms to estimate the potential impact of training data on a model’s future performance before the training process begins, which presents significant challenges. (2) Limitation in scalability: Traditional data valuation methods, developed for smaller models and datasets, often rely on computationally intensive techniques like leave-one-out training or influence functions Koh and Liang (2017) . These become prohibitively expensive or infeasible when applied to LLMs and their massive training datasets, due to non-linear scaling of computational requirements. (3) Data contamination concerns: There’s a scarcity of quantitative methods to evaluate how data valuation approaches help LLMs detect valuable, unseen information. Current methods often focus on fine-tuning well-trained LLMs, raising concerns about data contamination from pre-training exposure. Our Proposal: Learning to Approximate Linearized Future Influence Kernel (LinFiK). In this paper, we propose a novel approach to perform data valuation for LLMs: the Linearized Future Influence Kernel (LinFiK). LinFiK assesses each data point’s potential impact on the model’s final performance using first-order approximations of training dynamics. To address the scalability issue, we introduce ALinFiK, a leaning-based strategy to approximate LinFiK scores with significantly reduced computational overhead. Our approach not only provides a robust mechanism for third-party data valuation but also facilitates fair compensation for data providers based on the predicted value of their contributions. Our Contributions are threefold: • We introduce Linearized Future Influence Kernel (LinFiK) and provide a rigorous analysis of its stability throughout training. It enables efficient early-stage data selection, significantly enhancing model training efficiency. • To address scalability challenges, we develop an innovative distillation method to approximate LinFiK using significantly smaller models. This technique shows significant improvement in GPU memory, storage, and time consumption. • We designed a new dataset to quantify the ability of third-party data valuation algorithms to help select valuable information that the LLM has not encountered before, thus reducing potential concerns about data contamination. Through extensive experiments, we demonstrate that our method significantly outperforms existing data valuation approaches in terms of scalability, memory efficiency, and computational cost, particularly when applied to large LLMs. This work paves the way for more efficient model training and a more equitable data marketplace. 2 Background Figure 1: Integration of ALinFiK into Scalable Third-Party Data Valuation System of LLMs. LLM takes the task-specific test data and the sampled training data to produce LinFiK scores based on Equation 3 . The ALinFiK algorithm is then adopted to approximate LinFiK. This system satisfies the requirements of both model owners and data providers. (a) For model owners: the ALinFiK scores enable the selection of high-value training data that aligns with model training objectives; (b) For data providers: the ALinFiK scores provide transparent, quantitative metrics for fair data compensation. 2.1 Data Selection Given a large training set X=\{x_{i}\}_{i=1}^{N} , a task-specific test set Y=\{y_{j}\}_{j=1}^{M} , and a model with parameters w to be optimized, the data selection process is to identify a subset S from X so that the model performs optimally on Y after being trained on S ; This process can be formulated as an optimization problem: S^{*}={\arg\min}_{S\subset X}\mathcal{L}(Y,w_{S}) , where S^{*} is the optimal subset selected from X ; w_{S} represents the model parameters after being trained with selected training set S ; And \mathcal{L} is the loss function. 2.2 Data Influence Kernel We employ the concept of data influence to quantify the value of data samples on LLM. Data samples that exhibit a higher influence on the model are considered more valuable. We use a kernel \mathcal{K} to quantify the data influence. This allows us to transform the data selection problem into measuring the influence kernel \mathcal{K}(Y,x_{i}) of each training data sample x_{i} on the task-specific test dataset Y . From the model owner’s perspective, when selecting K data samples from the training dataset, the problem can be formulated as identifying the K samples with the highest influence: S^{*}={\arg\mathsf{TopK}}_{x_{i}\in X},\mathcal{K}(Y,x_{i}) . For the data providers, compensation can be allocated proportionally based on the influence of each data sample: \mathsf{Compensation}(x_{i})=\frac{\mathcal{K}(Y,x_{i})}{\sum_{i=1}^{K}\mathcal{K}(Y,x_{i})}*\mathsf{TotalCompensation} 2.3 Predicting Future Data Influence Let w_{t} and w_{T} denote the model’s early and final states during training. Most current data valuation algorithms Lin et al. (2024a) ; Choe et al. (2024) assess a given data sample’s value at w_{T} by tracking its impact during training. These methods have practical applications, but they fail to satisfy the needs of the model owners and data providers described earlier. In order to solve this problem, our objective is to estimate the value of each data sample at very early stage of the training, i.e. at w_{t} . This

Read the complete text