Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation

Jon-Paul Cacioli · 2026

Read Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation by Jon-Paul Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27405.txt.

Description

Cacioli adapts the Reliable Change Index from clinical psychology to compare Llama and Qwen versions item by item on 2,000 MMLU-Pro questions. Most items show no reliable change in either pair, yet analysable items move both ways, with gains partly cancelled by losses. Low-accuracy items tend to improve while high-accuracy items worsen, and Llama loses physics while Qwen loses law. Greedy single-shot evaluation misses 42 percent of changed items and falsely flags 25 percent of unchanged ones, so he recommends reporting churn with aggregate accuracy.

Catalog record

CREATORJon-Paul Cacioli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23420
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27405

Browse the full library · About the Tower of Babel