Do Data Valuations Make Good Data Prices?

Dongyang Fan, Tyler J. Rotello, Sai Praneeth Karimireddy · 2025

Read Do Data Valuations Make Good Data Prices? by Dongyang Fan, Tyler J. Rotello, Sai Praneeth Karimireddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05563.txt.

Introduction

Dongyang Fan and co-authors ask if data scores make fair data prices in a 2025 paper. Large language models lean on data from many owners, who should be paid. Scores rate how much each data set helps the model.

The team show that scores like leave one out and Data Shapley make poor pay rules. Owners can gain by lying about their costs, so the market wastes funds. Classic pay rules from market design fix that flaw. One rule gives the least pay that still draws true cost reports. The paper sets terms where both sides gain and the market runs well.

Catalog record

CREATORDongyang Fan, Tyler J. Rotello, Sai Praneeth Karimireddy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Data Valuations Make Good Data Prices? Dongyang Fan, Tyler J. Rotello, Sai Praneeth Karimireddy Abstract As large language models increasingly rely on external data sources, compensating data contributors has become a central concern. But how should these payments be devised? We revisit data valuations from a market-design perspective where payments serve to compensate data owners for the private heterogeneous costs they incur for collecting and sharing data. We show that popular valuation methods—such as Leave-One-Out and Data Shapley—make for poor payments. They fail to ensure truthful reporting of the costs, leading to inefficient market outcomes. To address this, we adapt well-established payment rules from mechanism design, namely Myerson and Vickrey-Clarke-Groves (VCG), to the data market setting. We show that Myerson payment is the minimal truthful mechanism, optimal from the buyer’s perspective. Additionally, we identify a condition under which both data buyers and sellers are utility-satisfied, and the market achieves efficiency. Our findings highlight the importance of incorporating incentive compatibility into data valuation design, paving the way for more robust and efficient data markets. Our data market framework is readily applicable to real-world scenarios. We illustrate this with simulations of contributor compensation in an LLM based retrieval-augmented generation (RAG) marketplace tasked with challenging medical question answering. 1 Introduction The emergence of large language models (LLMs) has placed data at the heart of technological and societal advancement. As concerns mount that the availability of data may not keep pace with the rapid growth of model sizes ( Villalobos et al., 2024 ) , the ability to source high-quality data has become a critical factor for the success of LLM companies. Currently, web-scale data crawling is conducted with little regard for data provenance, often leading to copyright infringement that impacts content owners. This has resulted in a growing number of copyright lawsuits, such as those documented by New York Times v. OpenAI (2023) ; Concord Music Group v. Anthropic (2023) . Figure 1: Data sources have different private unit costs Increasingly, content creators are choosing to opt out of contributing their work for AI training ( Longpre et al., 2024 ; Fan et al., 2025 ) . To encourage participation from data owners, there is a rising need to design an efficient data-trading market. Data owners should be fairly compensated for the use of their data, taking into account factors like the intrinsic cost of data generation, for example, it is more costly to gather expert-curated data than synthetic data (Figure 1 ). However, these costs are private - only the data collectors themselves know the effort and cost they put in. A well designed data marketplace could nevertheless discover these heterogeneous costs, match high-cost data collectors only with buyers who derive a high-value from the dataset, and extract appropriately high compensation. It is unclear if existing approaches to data pricing achieve such goals. Existing data valuation methods for machine learning primarily focus on interpretability and guiding data collection, often neglecting market dynamics. In a market scenario, data owners (sellers) can misreport data-related costs to maximize their compensation, while data buyers strive to improve model performance at minimal cost. When valuation methods such as Leave-One-Out ( Weisberg & Cook, 1982 ) and Data Shapley ( Ghorbani & Zou, 2019 ) are directly used as pricing rules in data markets, we point out that data sellers are incentivized to misreport their true costs, resulting in inefficient data market collaboration. To address the challenge of untruthful reporting, we draw on well-established payment mechanisms from game theory, specifically, the Myerson payment rule ( Myerson, 1981 ) and the Vickrey–Clarke–Groves (VCG) mechanism ( Vickrey, 1961 ; Clarke, 1971 ; Groves, 1973 ) , and adapt them to our data trading framework. Our analysis shows that: (1) the Myerson payment rule yields the minimum possible payment, making it optimal from the buyer’s standpoint; and (2) when allocations are made to maximize overall market welfare in an unconstrained setting, the VCG and Myerson payments coincide. Additionally, we demonstrate that when buyers’ utility functions are subadditive, total payments can be distributed across buyers while maintaining individual rationality for all participants. These findings highlight the crucial role of game-theoretic principles in designing data market payment schemes. Our data market modeling framework can be readily extended to real-world applications, ranging from simple mean estimation to optimizing data mixtures for LLM pretraining, to compensating content contributors in retrieval-augmented generation (RAG) systems. For the RAG market, the payments can be easily calculated by modifying the existing post-retrieval reranking process, showing great potential for seamless integration into current pipelines while enabling fair and truthful compensation for document owners. 2 Related Work Data Valuation Methods. Data valuation has gained growing popularity in machine learning applications, mainly for the purposes of explainability and addressing algorithmic fairness ( Pruthi et al., 2020 ; Liang et al., 2022 ) , guiding high-quality data selection ( Chhabra et al., 2024 ; Yu et al., 2024 ) . Among them there are two primary categories methods – Shapley value based and Leave-One-Out (LOO) based. Shapley value based methods include Data Shapley ( Ghorbani & Zou, 2019 ) and computationally feasible variations ( Jia et al., 2020 ; Wang et al., 2024 ) . LOO-based methods often encompass model retraining with one sample left out ( Koh & Liang, 2020 ) . In reality, rational participants in a market setting are likely to act strategically to maximize their profits. As a result, applying these pricing methods can lead to misreporting and inefficiencies within data markets. Auction Theory and Mechanism Design. Starting from the Arrow–Debreu model ( Arrow & Debreu, 1954 ) , a central question of market design is to set prices such that the net welfare of all participants is maximized. Such a market is said to be efficient . Auction theory ( Krishna, 2003 ) , and in general mechanism design, investigates how to how to design prices which preserve efficiency even if some information is unknown i.e. private . In the context of data markets, this implies that we want to design prices such that all participants truthfully report their true costs of data sharing and benefits received from said data, so that we can maximize social welfare. Data Markets. Due to unknown data sharing costs incurred by the data sellers, designing an efficient mechanism can be challenging. Dütting et al. (2021) show that with one data sample from the seller’s distribution, truthful mechanisms can be achieved with approximate market efficiency, assuming unit supply sellers. Rasouli & Jordarn (2021) design a truthful mechanism where data quality can be exchanged with monetary payments. However, they do not analyze its social efficiency. Agarwal et al. (2024) study truthful mechanism design when buyers’ externalities due to competition are known, aiming at maximizing social welfare or revenue. We refer to a recent survey ( Zhang et al., 2024 ) for a more detailed overview of the area. We note that most of these works often assume known simple structured valuation functions and combinatorial allocations, not suitable for machine learning worlds where the information sharing can happen in continuous space and the buyers’ valuations are directly connected to model performances, which is our focus. 3 What is a Desired Payment? 3.1 Our Proposed

Read the complete text