Accounting for Underspecification in Statistical Claims of Model Superiority
Thomas Sanchez, Pedro Macias Gordaliza, Meritxell Bach Cuadra, [2mm] CIBM Center for Biomedical Imaging · 2025
Read Accounting for Underspecification in Statistical Claims of Model Superiority by Thomas Sanchez, Pedro Macias Gordaliza, Meritxell Bach Cuadra, [2mm] CIBM Center for Biomedical Imaging (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02453.txt.
Description
Thomas Sanchez, Pedro Gordaliza, and Meritxell Bach Cuadra warn that many claimed improvements of medical imaging AI over rivals may be statistical mirages, and that the mirage is worse than previously calculated. Small performance gains are already likely to be false positives. The missing ingredient in earlier analyses is underspecification: models with nearly identical validation scores can behave differently on new data simply because of random initialisation or training dynamics.
The authors extend a recent statistical framework for false outperformance claims by adding underspecification as an extra source of variance. Their simulations show that even modest seed-to-seed variability, around one percent, substantially raises the amount of evidence needed before a superiority claim is justified. The practical message for medical imaging is blunt: report and model your training variance, or small winning margins mean little.
Catalog record
| CREATOR | Thomas Sanchez, Pedro Macias Gordaliza, Meritxell Bach Cuadra, [2mm] CIBM Center for Biomedical Imaging |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02453