Pitfalls of Conformal Predictions for Medical Image Classification
Hendrik Mehrtens, Tabea-Clara Bucher, Titus J. Brinker · 2025
Read Pitfalls of Conformal Predictions for Medical Image Classification by Hendrik Mehrtens, Tabea-Clara Bucher, Titus J. Brinker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18162.txt.
Introduction
Hendrik Mehrtens, Tabea-Clara Bucher and Titus Brinker warn that conformal prediction, a popular way to attach uncertainty guarantees to medical image classifiers, has sharp limits in 2025. The method promises calibrated confidence, which sounds ideal for safety-critical diagnosis.
Using examples from dermatology and histopathology, they show the guarantees break when the input data or label mix shifts, as happens between hospitals. The method also cannot be used to cherry-pick predictions for higher accuracy, and its promises do not hold for single classes or patient subgroups. With few classes, the output sets are often too coarse to help. Practitioners, they argue, should know these limits before trusting the numbers.
Catalog record
| CREATOR | Hendrik Mehrtens, Tabea-Clara Bucher, Titus J. Brinker |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18162
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pitfalls of Conformal Predictions for Medical Image Classification Hendrik Mehrtens, Tabea-Clara Bucher, Titus J. Brinker Abstract Reliable uncertainty estimation is one of the major challenges for medical classification tasks. While many approaches have been proposed, recently the statistical framework of conformal predictions has gained a lot of attention, due to its ability to provide provable calibration guarantees. Nonetheless, the application of conformal predictions in safety-critical areas such as medicine comes with pitfalls, limitations and assumptions that practitioners need to be aware of. We demonstrate through examples from dermatology and histopathology that conformal predictions are unreliable under distributional shifts in input and label variables. Additionally, conformal predictions should not be used for selecting predictions to improve accuracy and are not reliable for subsets of the data, such as individual classes or patient attributes. Moreover, in classification settings with a small number of classes, which are common in medical image classification tasks, conformal predictions have limited practical value. Keywords Conformal Predictions ⋅ \cdot Uncertainty Estimation 1 Introduction In recent years, deep learning has gained popularity in the medical domain. Nonetheless, the adaptation into clinical practice remains low, as current machine learning approaches based on deep neural networks are not able to provide reliable uncertainty estimates [ 1 , 2 , 3 ] , which however are a necessary condition for the safe and reliable operation in and admission to clinical practice. Over the last years many publications in the machine learning literature have been focused on estimating the predictive uncertainty of machine learning models using a plethora of approaches, however, recent publications [ 4 , 5 ] have shown that these methods do not consistently outperform the baseline of the confidence value of a single neural network when evaluated over a range of tasks. Moreover, these approaches are heuristic and lack calibration guarantees , which are essential in high-risk environments like medical classification. Conformal Predictions (CP) [ 6 ] have emerged as a promising framework for estimating uncertainty in neural network predictions, as they can provide a guaranteed level of true label coverage by constructing sets of predicted classes instead of single point predictions with uncertainty values, whereby larger sets indicate higher uncertainty. In this publication, we explore the viability and limitations of CP for medical image classification, where it has been used before, e.g. [ 7 , 8 , 9 , 10 , 11 ] , addressing common misconceptions and pitfalls, underpinning our observations with experiments conducted in histopathological tissue and dermatological skin classification tasks. While some of these limitations are well-known in the CP literature, e.g. [ 12 ] [ 9 ] , the intend of this publication is to serve as a comprehensive guide for future practitioners that might aspire to use or learn about conformal predictions for their own work in medical image classification, to avoid misconceptions and pitfalls, that we encountered ourselves, in discussions with colleagues and found in applications in the literature. After briefly introducing the concept of CP, we focus on the guarantees offered by conformal predictions and the challenges they pose in medical scenarios, particularly in dealing with domain shifts. Another challenge is maintaining coverage guarantees under shifts in the label distribution, which are harder to control and observe compared to changes in the input domain. We discuss the distinction between marginal and conditional coverage and its consequences for the coverage of predictive set sizes, coverage of individual classes, and other data subgroups. In this context, we also explore the connection between conformal predictions and selective classification [ 13 ] and why CP are insufficient for this task. Finally, we discuss the applicability and limitations of conformal predictions for medical image classifications, as here often tasks are considered that only have few classes or are even binary, for example, classification of benign and malignant tissue samples or the detection of a disease. 2 Conformal Predictions Conformal predictions (CP) [ 6 ] is a low assumption posthoc calibration approach for probabilistic classification models. Given a classifier \hat{Y} , that was already optimized on a training dataset D_{train} , and a calibration dataset D_{cal} , CP forms a set of classes C(\hat{Y}(x_{i})) out of the predictions of the model \hat{Y}(x_{i}) , so that \mathbb{P}_{(x_{i},y_{i})\in D_{test}}(y_{i}\in C(\hat{Y}(x_{i})))\geq 1-\alpha\;, (1) which is called the marginal coverage guarantee, where 1-\alpha is the desired coverage and the set-valued function C is found on the calibration set. Conformal predictions make no assumptions on the model or the data but only necessitate the property of exchangeability between the datasets D_{cal} and D_{test} . The set-valued function C that forms predictive sets over the output of the classifier \hat{Y} is found on the held out calibration data set D_{cal} using a non-parametric quantile-based algorithm. Given the exchangeability of both data collections, the properties of this function then also hold for the test data set. Various approaches exist for forming the prediction sets using different conformity scores. In our experiments, we employ the adaptive prediction sets (APS) formulation [ 14 ] , previously used in image classification [ 15 ] . While these approaches may employ different conformity scores to construct the predictive sets C(\hat{Y}(x_{i})) , they share the common goal of taking an initially uncalibrated uncertainty estimate from the trained classifier \hat{Y} and finding a set of classes for each prediction that satisfies the expected marginal coverage guarantee based on the classifier’s behavior on the calibration dataset and are by that nature susceptible to the in the following discussed pitfalls and limitations. 3 Conformal Prediction for histopathology and dermatology Dataset CAMELYON17 HAM10K Num. Classes 2 7 Validation Set 126672 967 Calibration Set 1000 500 Accuracy (ID) 97.56\%_{\pm 0.1} 77.76\%_{\pm 0.4} Table 1: Overview of the used dataset, number of calibration points and the reached in-distribution accuracy over 5 evaluations To showcase our observations we first trained five ResNet-34 [ 16 ] on two medical image classification datasets each: HAM10K [ 17 ] for multi-class skin-lesion classification and CAMELYON17 [ 18 ] for histo- pathological whole-slide image (WSI) classification. The HAM10K dataset consists of seven skin conditions, while the CAMELYON17 dataset includes 50 WSIs with annotated tumor regions from five different clinics. We preprocessed the WSIs into non-overlapping patches of size 256x256 pixels. Our training setup followed the approach in [ 5 ] , using the same augmentations and hyperparameters for both datasets. We trained our neural networks on Centers 1, 3, and 5 of the CAMELYON17 dataset and produced predictions for the Centers 2 and 4 for domain-shift experiments. It’s important to note that conformal predictions have been previously employed in histopathology [ 7 , 8 ] and dermatology [ 9 ] . Our goal with these experiments is to showcase the applicability and limitations of conformal predictions for medical classification tasks, rather than achieving state-of-the-art classification results. We used singular neural networks for uncertainty estimation, as they have been shown to be sufficiently competitive [ 4 , 5 ] to more advanced approaches like Deep Ensembles [ 19 ] or Monte-Carlo Dropout [ 20 , 21 ] . Table 1 provides details of the datasets and the achieved accuracies. We utilized the APS algorithm [ 14 , 15 ] to compute the conformal prediction sets. We used 1000 calibration data points for