Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals
Pascaline André ∗ \ast, Charles Heitz ∗ \ast, Evangelia Christodoulou, Annika Reinke, Carole H. Sudre, Michela Antonelli, Patrick Godau, M. Jorge Cardoso, Antoine Gilson, Sophie Tezenas du Montcel, Gaël Varoquaux ⋆ \star ,, Lena Maier-Hein ⋆ \star, Olivier Colliot ⋆ \star · 2026
Read Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals by Pascaline André ∗ \ast, Charles Heitz ∗ \ast, Evangelia Christodoulou, Annika Reinke, Carole H. Sudre, Michela Antonelli, Patrick Godau, M. Jorge Cardoso, Antoine Gilson, Sophie Tezenas du Montcel, Gaël Varoquaux ⋆ \star ,, Lena Maier-Hein ⋆ \star, Olivier Colliot ⋆ \star (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17103.txt.
Introduction
Pascaline André and a large team check how firm scan AI scores truly are. A score with no range can look more firm than it is. Ranges, called confidence intervals, show that doubt. Their 2026 preprint runs a large-scale check of those ranges. It spans many scan tasks and AI tools. The aim is sound doubt marks for care use.
Hospitals need doubt marks before they trust a tool. A tool that tops a chart may tie with the next in truth. Wide ranges flag scores that need more data. The check sets a base for fair score claims. It aids rules for AI in clinics.
Catalog record
| CREATOR | Pascaline André ∗ \ast, Charles Heitz ∗ \ast, Evangelia Christodoulou, Annika Reinke, Carole H. Sudre, Michela Antonelli, Patrick Godau, M. Jorge Cardoso, Antoine Gilson, Sophie Tezenas du Montcel, Gaël Varoquaux ⋆ \star ,, Lena Maier-Hein ⋆ \star, Olivier Colliot ⋆ \star |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 124 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17103
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals Pascaline André ∗ \ast, Charles Heitz ∗ \ast, Evangelia Christodoulou, Annika Reinke, Carole H. Sudre, Michela Antonelli, Patrick Godau, M. Jorge Cardoso, Antoine Gilson, Sophie Tezenas du Montcel, Gaël Varoquaux ⋆ \star ,, Lena Maier-Hein ⋆ \star, Olivier Colliot ⋆ \star Abstract Performance uncertainty quantification is essential for reliable validation and eventual clinical translation of medical imaging artificial intelligence (AI). Confidence intervals (CIs) play a central role in this process by indicating how precise a reported performance estimate is. Yet, due to the limited amount of work examining CI behavior in medical imaging, the community remains largely unaware of how many diverse CI methods exist and how they behave in specific settings. The purpose of this study is to close this gap. To this end, we conducted a large-scale empirical analysis across a total of 24 segmentation and classification tasks, using 19 trained models per task group, a broad spectrum of commonly used performance metrics, multiple aggregation strategies, and several widely adopted CI methods. Reliability (coverage) and precision (width) of each CI method were estimated across all settings to characterize their dependence on study characteristics. Our analysis revealed five principal findings: 1) the sample size required for reliable CIs varies from a few dozens to several thousands of cases depending on study parameters; 2) CI behavior is strongly affected by the choice of performance metric; 3) aggregation strategy substantially influences the reliability of CIs, e.g. they require more observations for macro than for micro; 4) the machine learning problem (segmentation versus classification) modulates these effects; 5) different CI methods are not equally reliable and precise depending on the use case. These results form key components for the development of future guidelines on reporting performance uncertainty in medical imaging AI. 1 1 footnotetext: These authors contributed equally as first authors. 2 2 footnotetext: These authors contributed equally as senior authors. 3 3 footnotetext: Correspondence to: [email protected] and [email protected] . Disclaimer: this is a working paper, and is still work in progress. Please do not hesitate to provide comments or point towards missing references to [email protected] and [email protected] . Keywords: Medical imaging, Validation, Confidence intervals, Segmentation, Classification, Uncertainty quantification, Trustworthy machine learning 1. Introduction Over the past decade, there has been an exponential increase in the number of papers on machine learning (ML) for medical imaging ( 86 ) . Despite this rapid growth, there is still little translation from the literature into routine clinical practice. One hindrance to translation is the lack of uncertainty quantification of performance estimates in research papers ( 23 ) . Indeed, performance estimates are computed from test sets of finite size and are therefore inherently subject to sampling variability. As a result, these estimates should be interpreted together with measures of uncertainty. Confidence intervals (CIs) are arguably the most common way to provide a measure of the uncertainty of the reported performance estimate. As a matter of fact, the U.S. Food and Drug Administration (FDA) requires companies to report confidence intervals along with performance estimates when submitting Artificial Intelligence (AI)-enabled medical devices for regulatory approval ( 39 ) . However, estimating uncertainty through CIs is not a trivial task. Different CI methods exist and the choice of an inappropriate method can lead to either overconfidence or underconfidence in the estimated model performance (Figure 1 ). In particular, methods producing too narrow intervals often lead to overconfidence, as the proposed interval might miss the true performance of the model. Conversely, methods producing too wide intervals are underconfident as they do not manage to locate the true value with enough precision. A good confidence interval method should therefore balance these two aspects by accurately identifying the range of plausible values for the performance estimate while providing an informative assessment of its precision. In other words, a CI method has two key characteristics: reliability and precision. Reliability means that the method most often catches the true value. Precision means that the CI width is as small as possible. Despite their importance, there are very few papers on CIs in medical imaging AI ( 36 ; 80 ) and their scope remains narrow, typically considering only a small set of models and performance metrics. While related work exists in generalist AI ( 65 ) and in specific fields, e.g. psychology ( 74 ) and economics ( 21 ) , they either rely on parametric simulations or study specific statistical regimes that may differ from medical image analysis. Medical imaging AI exhibits domain-specific characteristics that can strongly influence the behavior of confidence interval (CI) methods. In particular, a wide range of task- and application-specific performance metrics are required and guidelines have been established to support appropriate metric selection ( 61 ) . In addition, medical imaging studies often rely on small to moderate sample sizes compared to those typically used in general ML or computer vision ( 22 ) . In this work, we present a large-scale investigation of the behavior of widely used CI methods in the context of medical imaging. Specifically, our central research question was: how are reliability and precision of CI methods affected by study characteristics including ML problem (classification, segmentation), sample size, performance metrics and aggregation method (Figure 2 , left panel)? Our main contributions are the following: • We performed an extensive experimental study of various segmentation and classification tasks (12 for each) and trained models (19 for each learning problem) corresponding to 456 ( 2\times 12\times 19 ) benchmark instances covering a variety of settings. • Across the 456 benchmark instances, we systematically studied the reliability (coverage) and precision (width) of different CI methods (parametric and non-parametric) for different performance metrics, aggregation methods and sample sizes. • We unveiled key relationships between CI behavior and study characteristics (Figure 2 , right panel). Figure 1 : Not all confidence interval (CI) methods can be universally trusted. Here, CI 2 is too wide and thus underconfident about the target value. CI 3 is unreliable as it completely misses the target value. CI 1 is both reliable and precise. Figure 2 : Link between study characteristics and behavior of confidence intervals (CIs). Left panel: main characteristics of a medical imaging study. Right panel: key findings of our investigation, revealing how study characteristics impact the reliability and precision of CIs. A preliminary version of this work appeared in the proceedings of the MICCAI BRIDGE Workshop 2025 ( 8 ) . The present journal version considerably expands upon this by considering: 1) not only segmentation but also classification; 2) 14 performance metrics instead of 2; 3) 5 summary statistics instead of 2; 4) not only the coverage but also the width of the CIs. Such combinations allow us to study CI behavior in 108 different configurations versus 4 in the workshop paper. 2. Background and related work This section first provides a brief, non-technical, overview of the concepts used in this paper, including metrics, summary statistics and confidence intervals (Section 2.1 ). This part can be skipped by readers who are already familiar with these notions. Readers who prefer a more formal treatment can refer to Supplementary Sections S1 and S2 . Section 2.2 then describes related