When LLMs get significantly worse: A statistical approach to detect model degradations

Jonas Kübler, Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin · 2026

Read When LLMs get significantly worse: A statistical approach to detect model degradations by Jonas Kübler, Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10144.txt.

Description

Even theoretically lossless optimizations can shift a model's outputs through numerical error, so tiny accuracy drops may or may not signal real degradation. A 2026 paper led by Jonas Kübler, with six co-authors, proposes a statistically sound testing framework based on McNemar's test to tell the difference. The key insight is to confront model scores sample by sample rather than aggregating at task level.

The framework controls false positives while detecting degradations efficiently, and it offers three ways to aggregate accuracy across benchmarks into one decision. Built on the widely used LM Evaluation Harness, it flags genuinely degraded models while sparing provably lossless optimizations. Even accuracy drops of three-tenths of a percent can be confidently attributed to real degradation rather than noise.

Catalog record

CREATORJonas Kübler, Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-10144

Browse the full library · About the Tower of Babel