Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking
Solomon Messing · 2026
Read Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking by Solomon Messing (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11581.txt.
Description
Messing shows measurement error in LLM pipelines distorts annotation, tests, and benchmarking. Confidence intervals ignore spread from judge choice, temperature, and prompt phrasing, producing gaps that worsen with more data. He splits uncertainty into sources, separates variance shrinking with data from swings with design choices, and uses projections to cut error. Naive standard errors run 40 to 60 percent smaller than corrected ones, and 95 percent coverage falls as data grows while corrected coverage holds. TEE-guided pipelines shrink the gaming surface from 56 to 32 Elo, and acting on projections halves MMLU error.
Catalog record
| CREATOR | Solomon Messing |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 162061 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11581