Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation
Jon-Paul Cacioli · 2026
Read Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation by Jon-Paul Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27405.txt.
Description
Cacioli adapts the Reliable Change Index from clinical psychology to compare Llama and Qwen versions item by item on 2,000 MMLU-Pro questions. Most items show no reliable change in either pair, yet analysable items move both ways, with gains partly cancelled by losses. Low-accuracy items tend to improve while high-accuracy items worsen, and Llama loses physics while Qwen loses law. Greedy single-shot evaluation misses 42 percent of changed items and falsely flags 25 percent of unchanged ones, so he recommends reporting churn with aggregate accuracy.
Catalog record
| CREATOR | Jon-Paul Cacioli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23420 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27405