Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking

Solomon Messing · 2026

Read Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking by Solomon Messing (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11581.txt.

Description

Messing shows measurement error in LLM pipelines distorts annotation, tests, and benchmarking. Confidence intervals ignore spread from judge choice, temperature, and prompt phrasing, producing gaps that worsen with more data. He splits uncertainty into sources, separates variance shrinking with data from swings with design choices, and uses projections to cut error. Naive standard errors run 40 to 60 percent smaller than corrected ones, and 95 percent coverage falls as data grows while corrected coverage holds. TEE-guided pipelines shrink the gaming surface from 56 to 32 Elo, and acting on projections halves MMLU error.

Catalog record

CREATORSolomon Messing
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE162061
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11581

Browse the full library · About the Tower of Babel