Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 35
Mélissa Tamine, Otmane Sakhi, Benjamin Heymann · 2025
Read Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 35 by Mélissa Tamine, Otmane Sakhi, Benjamin Heymann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15765.txt.
Introduction
Melissa Tamine and co authors make Shapley based data valuation practical for language model alignment in a 2025 paper, dated December 12. When a model is aligned on several data sources, Shapley values can score each source's contribution. The standard method needs a separate alignment for every coalition of sources, an exponential cost.
For direct alignment algorithms, the authors show that sequential training, source by source, makes each stage add a known amount to response log probabilities. Any coalition's score can then be rebuilt from the base model and one model per source, cutting cost from exponential to linear. Experiments across base models and reward models show how source contributions vary by evaluation criteria.
Catalog record
| CREATOR | Mélissa Tamine, Otmane Sakhi, Benjamin Heymann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15765
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 35 Mélissa Tamine, Otmane Sakhi, Benjamin Heymann Abstract . Data is a critical asset for training large language models (LLMs), alongside compute resources and skilled workers. While some training data is publicly available, substantial investment is required to generate proprietary datasets, such as human preference annotations or to curate new ones from existing sources. As larger datasets generally yield better model performance, two natural questions arise. First, how can data owners make informed decisions about curation strategies and data sources investment? Second, how can multiple data owners collaboratively pool their resources to train superior models while fairly distributing the benefits? This problem, data valuation , which is not specific to large language models, has been addressed by the machine learning community through the lens of cooperative game theory, with the Shapley value being the prevalent solution concept. However, computing Shapley values is notoriously expensive for data valuation, typically requiring numerous model retrainings, which can become prohibitive for large machine learning models. In this work, we demonstrate that this computational challenge is dramatically simplified for LLMs trained with Direct Preference Optimization (DPO). We show how the specific mathematical structure of DPO enables scalable Shapley value computation. We believe this observation unlocks many applications at the intersection of data valuation and large language models. C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction Large language models (LLMs) are the result of collaborative training and alignment pipelines: a single deployed model may have been pre-trained on heterogeneous web-scale corpora, often mixed with proprietary data sources, and then adapted through instruction tuning ( 30 ) and several stages of preference-based alignment, such as Reinforcement Learning from Human Preferences (RLHF) ( 6 ; 45 ; 18 ) typically implemented with policy-gradient methods like Proximal Policy Optimization (PPO) ( 33 ) , and more recent methods such as Direct Preference Optimization (DPO) ( 31 ) or weak-to-strong supervision schemes ( 4 ) . Recent scaling-law studies show that, beyond architecture, LLMs’ performance is mainly driven by the amount, diversity, and quality of training data ( 21 ; 15 ) . This reflects the adage that data is the new oil ( 19 ) . The key scarce resource of LLMs is not the model design, but the data provided by companies, institutions, and user communities: people are paid to label and rank model outputs, LLMs themselves are used to generate additional training data, and companies engage in legal battles over access to valuable corpora. This raises a central data valuation question: how should we attribute the contribution of each data source to the final behavior of an LLM? Answering this question is not just of academic interest: data valuation (i.e., a systematic way to attribute value to a data source) is a prerequisite for data markets ( what is a fair price for a dataset? ) ( 1 ; 5 ) , contractual guarantees ( what level of performance can we promise to a contributor? ), incentive design ( how should we reward agents whose data improves alignment? ) ( 22 ; 35 ) , and even basic notions of responsibility ( which data source made the model toxic on an input? ) ( 12 ; 19 ; 20 ; 38 ) . Cooperative game theory, in particular the Shapley value ( 34 ) , provides a formalism for such data valuation problems: interpret each data source as a player, define the utility of any coalition of players, and use the Shapley value to fairly split the utility among all players. However, directly instantiating this paradigm for LLM fine-tuning is computationally prohibitive. In Shapley-based data valuation, the core bottleneck is that the utility must be evaluated for every coalition of data sources, and the number of such coalitions grows exponentially with the number of sources ( 34 ; 12 ) . Even in classical supervised learning, this already leads to an impractical number of model retrainings. Still, the situation is worse for LLMs. Each utility evaluation now requires fine-tuning a large model. With PPO or DPO, this means running a complete preference optimization loop on each coalition of data sources, even if one relies on emulation or distillation techniques that use a smaller model to approximate the effect of fine-tuning the LLM ( 27 ) (such methods may lower the cost per run, but they do not remove the exponential number of coalition-specific runs). The present proposal addresses this obstacle. We focus on preference-based fine-tuning, and in particular DPO, where the objective is defined directly over pairs of preferred and dispreferred responses ( 31 ) . Our key observation is that training sequentially across multiple datasets corresponds to summing the reward models learned from each dataset independently. Hence, in this setting, the utility of a coalition of datasets need not be defined by a dedicated fine-tuning, and we can take inspiration from recent work on language model arithmetic ( 9 ) : starting from one base model and a collection of models, each fine-tuned on a single dataset, one can construct at inference time composite models that approximately capture the effect of training on unions of datasets, by combining their output probabilities using a simple arithmetic rule over the fine-tuned models. Building on this observation, we propose a Shapley value approximation method for LLMs that reduces the number of required fine-tunings from exponential to linear in the number of data sources. We first use DPO to fine-tune one model per data source, then apply model arithmetic to construct, at inference time, an approximate model for any coalition of sources (a coalition model ). The utility of a coalition is then defined as the performance of the coalition model on a fixed evaluation task. This enables estimating the Shapley value for all data sources while performing only one fine-tuning per source. Related works. There are two streams of Shapley value applications for aligned LLMs: one focused on model explainability for token or feature-level attribution of predictions ( 41 ; 42 ; 16 ; 44 ) , and another focused on data valuation for quantifying training data contributions ( 32 ; 13 ; 40 ; 28 ) . Our work is orthogonal to the first stream and contributes to the second. Its novelty, compared to prior data valuation studies for LLMs, is to exploit the specific mathematical structure of preference-based alignment (DPO combined with language model arithmetic) to reduce the cost of computing coalition utilities. In this work, we show that combining DPO with the philosophy of language model arithmetic enables us to approximate coalition utilities without performing coalition-specific fine-tunings. {\lx@inpgf@ignorespaces{(\text{dataset collections},\cup)}} {\lx@inpgf@ignorespaces{(\text{reward models},+)}} {\lx@inpgf@ignorespaces{(\text{policies},*)}} Reward modeling DPO KL-regularized RL Figure 1. We can equip the collections of datasets, the set of reward models, and the set of policies with binary operators to give them a group (or semi-group) structure. After doing so, we observe that some training methods, such as sequential DPO, preserve this structure and therefore induce morphisms between those semi-groups. 2. Preliminary 2.1. Evaluation of LLMs We consider