Data Reliability Scoring

Yiling Chen, Shi Feng, Paul Kattuman, Fang-Yi Yu · 2025

Read Data Reliability Scoring by Yiling Chen, Shi Feng, Paul Kattuman, Fang-Yi Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17085.txt.

Introduction

Yiling Chen and three co-authors ask how to grade data trust with no ground truth in this 2025 paper. Data often come from sources that may bend what they report. The truth stays hidden, yet outcomes of an unknown test can be seen. The authors seek a score that ranks data sets by how far reports stray.

Their score uses the volume spanned by spread vectors from the data. They prove it keeps key true rankings of data set quality. Up to scale, it is the sole score that ignores which test was used. Tests on fake noise, image traits, and job data back the score. It gives a fair way to compare sources when truth stays hidden.

Catalog record

CREATORYiling Chen, Shi Feng, Paul Kattuman, Fang-Yi Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17085

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data Reliability Scoring Yiling Chen, Shi Feng, Paul Kattuman, Fang-Yi Yu Abstract How can we assess the reliability of a dataset without access to ground truth? We introduce the problem of reliability scoring for datasets collected from potentially strategic sources. The true data are unobserved, but we see outcomes of an unknown statistical experiment that depends on them. To benchmark reliability, we define ground-truth–based orderings that capture how much reported data deviate from the truth. We then propose the Gram determinant score , which measures the volume spanned by vectors describing the empirical distribution of the observed data and experiment outcomes. We show that this score preserves several ground-truth-based reliability orderings and, uniquely up to scaling, yields the same reliability ranking of datasets regardless of the experiment – a property we term experiment agnosticism. Experiments on synthetic noise models, CIFAR-10 embeddings, and real employment data demonstrate that the Gram determinant score effectively captures data quality across diverse observation processes. 1 Introduction Reliable data can effectively inform decision-making. For example, vehicle condition and driving behavior data help insurance companies set policies; investor’s positions guide regulators in adjusting financial market rules; and during the COVID-19 pandemic, case numbers were used by governments to allocate medical resources. Yet, such data are typically reported by people. They can be noisy, and more importantly, strategically or maliciously distorted. Direct verification is often impossible or impractical. This raises a central question: how can we tell whether a dataset is reliable? Answering this would greatly enhance the value of data-driven methods for decision-making. Without further knowledge, this question is unresolvable. But in practice, we often have access to data that are related to the private data in question. For instance, insurance company may use telematic devices–albeit imperfect–to estimate vehicle condition; regulators can observe trading volumes correlated with investors’ positions; and governments track COVID mortality numbers linked to true case counts through disease fatality rates. Such auxiliary observations can provide useful information to assess how well the reported data are consistent with the unobservable ground truth. In this paper, we initiate the study of reliability scoring for datasets collected from potentially strategic or noisy sources. Although the underlying truth remains unknown, we assume access to outcomes of unknown statistical experiments that depend on it. Our contributions include: \bullet We formalize the problem of reliability scoring from observations generated by unknown experiments. ( Section 2 ) \bullet We introduce ground-truth-based dataset reliability orderings as benchmarks for evaluating reliability scores. ( Section 2.3 ) \bullet We propose a novel reliability measure, the Gram Determinant Score , along with its kernel variant, which preserves several ground-truth-based dataset reliability orderings under certain conditions. Moreover, we show that the Gram Determinant Score is, up to scaling, the unique reliability score that produces the same dataset ranking for all experiments – a property we term experiment agnosticism . ( Section 4 ) \bullet We analyze the limitations of reliability scoring and show that the conditions under which the Gram Determinant Score preserves reliability orderings are nearly tight. ( Section 3 ) \bullet We empirically validated the Gram Determinant Score using synthetic data, CIFAR-10 image dataset, and employment data. ( Section 5 ) The Gram Determinant Score admits a geometric interpretation: it measures the volume of the parallelepiped spanned by the joint distribution of the reported data and the experiment outcomes. As the reported data deviate further from the truth, this volume decreases. ( Footnote 6 ) 1.1 Related Work Early frameworks categorize data reliability into intrinsic, contextual, accessibility, and representational dimensions. [ 44 , 40 ] Our work focuses on intrinsic reliability—the extent to which reported data match the true data—using auxiliary observations. Our approach is inspired by information elicitation, which designs scoring mechanisms that incentivize truthful reporting. A key distinction is our emphasis on preserving ordinal relationships: assigning higher scores to more reliable data. Traditional elicitation instead focuses solely on ensuring that truthful reporting is strictly optimal among alternatives. Information elicitation has two main settings (1) when the scoring mechanism can access the ground truth, e.g., proper scoring rules for predictions of future observable events [ 16 , 38 , 31 , 13 , 32 ] ; and (2) peer prediction mechanisms, which do not have access to ground truth but rely on multiple agents’ reports [ 35 , 12 ] . The most relevant work is Kong 2024 , which introduces determinant mutual information and inspires our Gram Determinant Score. We provide a more detailed comparison with Kong 2024 in the Appendix. A recent works use Shannon (pointwise) mutual information to evaluate dataset and introduce Blackwell ordering to compare reported dataset. [ 48 ] Traditional statistical approaches [ 21 , 34 ] often assess reliability under distributional assumptions. In contrast, our method evaluates reliability agnostic to the underlying distribution. There are several general-purpose score that measures the stochastic relationship between random variables, e.g., KL-divergence [ 30 ] , f -divergence [ 11 ] , determinant [ 50 , 45 ] , PCA [ 2 ] . But they often lack clear connections to standard, interpretable criteria such as accuracy or data integrity. On the other hand, one line of data valuation focus on task-dependent utility—quantifying the value of a dataset or individual sample with respect to a specific objective. Examples include value of information in decision theory [ 19 , 8 ] , influence-based valuation [ 9 , 26 ] , and data Shapley [ 14 ] . In contrast, our reliability scoring aims to evaluate datasets in a task-agnostic and experiment-agnostic manner. Other related areas include learning with noisy labels [ 37 ] , which typically assumes that reports are corrupted by independent noise. Some works (e.g., [ 33 ] ) relax this by allowing unknown noise, but our setting is more general: auxiliary observations may lie in an entirely different space. Anomaly detection [ 6 ] addresses distribution shifts, but focuses on adaptive detection rather than reliability scoring. Finally, reliability theory primarily studies system robustness to failure [ 15 ] , a concept distinct from data reliability. 2 Model In this section, we introduce the problem of designing data reliability scores to assess how much a dataset deviates from its inaccessible ground truth. To benchmark reliability, we propose ground-truth-based reliability orderings—partial orders over datasets that compare their relative deviations from the same true dataset. The ideal goal of a reliability score is to preserve these orderings, assigning higher scores to datasets that more faithfully reflect the true data. 2.1 Basic Setup There is a single data source (an agent) who has access to a set of true data {\bm{x}}=(x_{1},\dots,x_{N}) of size N . 1 1 1 {\bm{x}} is non-time-series data. Hence, the order of the data within the set is not important. The agent provides reported data \hat{{\bm{x}}}=(\hat{x}_{1},\dots,\hat{x}_{N}) , which can potentially be different from {\bm{x}} . Let \mathcal{X}=[d] be the set of d possible data values. Thus, x_{n}\in\mathcal{X} and \hat{x}_{n}\in\mathcal{X} for all n . Our goal is to evaluate how reliably the reported data \hat{{\bm{x}}} reflects the true data {\bm{x}} . Although {\bm{x}} is unobserved, we have access to additional observable data {\bm{y}}=(y_{1},\dots,y_{N}) , called observations , which are indirectly related to {\bm{x}} . The observation space \mathcal{Y} may

Read the complete text