The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains
Jung Min Kang · 2026
Read The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains by Jung Min Kang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11205.txt.
Description
Jung Min Kang shows a scaling law of evaluation failure: simple averaging breaks down when data is sparse and item difficulty gaps are wide, while item response theory finds ground truth. Benchmark scoring mostly uses the plain mean across test items. But rankings mislead when the test matrix is sparse and items vary a lot in hardness.
Through simulations across NLP/GLUE, clinical drug trials, self-driving vehicle safety, and cybersecurity product evaluation, Spearman rank correlation is tracked. Simple-average rankings versus ground truth fall from a perfect score with full data to about four-fifths at 67 percent data with high difficulty spread. A standard two-parameter logistic item response theory model holds near perfect in all cases. Testers get a law: mean accuracy declines with the product of thin data and the difficulty gap, while IRT stays robust.
Catalog record
| CREATOR | Jung Min Kang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41374 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11205