When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images
Loris Cino, Pier Luigi Mazzeo, Alessandro Martella, Giulia Radi, Renato Rossi, Cosimo Distante · 2026
Read When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images by Loris Cino, Pier Luigi Mazzeo, Alessandro Martella, Giulia Radi, Renato Rossi, Cosimo Distante (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00651.txt.
Introduction
Cino and colleagues study images where AI and experts fail together. Skin checks with convolutional networks show strong medical promise, yet most studies only rank models against human experts. This study instead probes the images alone. Testing several network designs, the team finds images wrongly labelled by every model, a pattern proven beyond random chance. To test whether failures come from model bias or from the images, expert dermatologists judged the hard cases beside a control group. Human scores fell on the images the AI got wrong.
Catalog record
| CREATOR | Loris Cino, Pier Luigi Mazzeo, Alessandro Martella, Giulia Radi, Renato Rossi, Cosimo Distante |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54298 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00651
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images Loris Cino, Pier Luigi Mazzeo, Alessandro Martella, Giulia Radi, Renato Rossi, Cosimo Distante Abstract The integration of artificial intelligence (AI), particularly Convolutional Neural Networks (CNNs), into dermatological diagnosis demonstrates substantial clinical potential. While existing literature predominantly benchmarks algorithmic performance against human experts, our study adopts a novel perspective by investigating the intrinsic complexity of dermatoscopic images Through rigorous experimentation with multiple CNN architectures, we isolated a subset of images systematically misclassified across all models—a phenomenon statistically proven to exceed random chance. To determine if these failures stem from algorithmic biases or inherent visual ambiguity, expert dermatologists independently evaluated these challenging cases alongside a control group. The results revealed a collapse in human diagnostic performance on the AI-misclassified images. First, agreement with ground-truth labels plummeted, with Cohen’s kappa dropping to a mere 0.08 for the difficult images, compared to a 0.61 for the control group. Second, we observed a severe deterioration in expert consensus; inter-rater reliability among physicians fell from moderate concordance (Fleiss’ kappa = 0.456) on control images to only modest agreement (Fleiss’ kappa = 0.275) on difficult cases. We identified image quality as a primary driver of these dual systematic failures. To promote transparency and reproducibility, all data, code, and trained models have been made publicly available. † † highlights: Multiple AI algorithms systematically misclassify the same images. Human diagnostic accuracy and consensus degrade on these AI-misclassified cases. Image artifacts and low quality partially explain these shared diagnostic errors. keywords Artificial Intelligence (AI) ,Machine Learning (ML) ,Convolutional Neural Networks (CNNs) ,Deep Learning ,Medical Image Analysis ,Dermatology ,Image Quality ,Statistical Analysis † † credit: Conceptualization, Methodology, Formal analysis, Writing – original draft † † credit: Methodology, Formal analysis, Writing – review & editing † † credit: Conceptualization, Methodology, Writing – original draft † † credit: Supervision, Project administration, Resources † † credit: Resources † † credit: Resources † † credit: Supervision, Project administration, Funding acquisition † † corresponding: Corresponding author 1 Introduction Skin tumors, including Basal Cell Carcinoma (BCC) [ 42 ] , Squamous Cell Carcinoma (SCC), and Melanoma (MEL), represent some of the most common and clinically significant dermatological malignancies worldwide. In light of the significant health risks they pose, dermatoscopy has emerged as a primary non-invasive technique for their early detection. This method is widely adopted in clinical practice, as it enhances the visualization of subsurface skin structures, supports more accurate diagnosis, and ultimately contributes to earlier intervention and reduced mortality. Despite dermatoscopy effectiveness, histopathological examination remains the gold standard for diagnosis, as the dermatoscopic characteristics assessed using the ABCDE (Asymmetry, Border, Color, Diameter, Evolution) rule [ 36 ] may not be decisive for a specific cutaneous neoplasm, especially in the case of difficult lesions that appear strikingly similar. In addition to its role in oncologic dermatology, dermatoscopy is extensively utilized in the evaluation of chronic inflammatory skin conditions, which affect approximately 25–30% of the population, including psoriasis. Furthermore, it is employed in the assessment of parasitic infestations and the examination of skin appendages, such as nails and hair [ 58 ] . As dermatoscopy is a fundamentally image-centric field, the application of deep neural networks—driven by the remarkable advancements in CNNs—represents a natural evolution in diagnostic analysis [ 39 ] . Over the years, various methodologies have been explored to integrate AI into dermatological diagnostics, encompassing both deep learning-based and alternative approaches. A seminal study demonstrated that CNN-based models could achieve diagnostic performance comparable to that of expert dermatologists, marking a significant step in AI-driven dermatology [ 18 ] . Other works demonstrated that performance can be enhanced by utilizing multiple models and aggregating their predictions to generate a more robust and reliable outcome [ 30 , 23 , 44 ] , a methodology commonly referred to as ensemble learning. Another strategy for improving model efficacy involves incorporating additional patient-specific information, such as age and the anatomical location of the lesion [ 32 ] . Notably, Gannour et al. [ 23 ] integrate both ensemble methods and individual patient data in their analysis. However, the extent to which these approaches—ensemble learning and patient-specific data—contribute to overall model improvement remains an open question. Furthermore, to the best of the authors’ knowledge, no prior work has employed a meta-classifier, a model fitted on other models predictions, CNNs in this case. Despite the wide literature deep learning is not widely adopted in the dermatoscopy [ 62 , 37 ] , an empirical evidence is that the United Stated Food and Drug Administration approved several AI algorithms for medical use but none of them regard dermatology [ 64 ] . The limited adoption of AI-based dermatological diagnostic tools in clinical practice can be attributed to several technical factors: (i) performance degradation when these algorithms are applied to OOD; (ii) issues related to data quality [ 62 ] ; and (iii) the unclear impact of AI-assisted diagnosis on clinical decision-making. While all these factors are essential for a comprehensive understanding of the literature, this study focuses primarily on the last two aspects. Specifically, we analyze the diagnostic accuracy of expert dermatologists and AI models at the image level, demonstrating that both human experts and AI systems struggle with the same subset of images. Furthermore, we identify a significant presence of low-quality images within commonly used dermatological datasets and propose an automated approach for detecting blurred images. 1.1 Out-of-distribution performance The investigation into the out-of-distribution performance of machine learning models applied in dermatology has garnered considerable attention in contemporary research [ 38 ] . Recent studies have highlighted a decline in model performance when evaluated on images that deviate from those included in the training dataset [ 62 ] . This decline is particularly pronounced when models are tested on individuals from ethnicities and with skin colors that are underrepresented in the training data, highlighting the need to test the model on different Fitzpatrick skin type labels [ 26 , 25 ] . Interestingly, this phenomenon is also observed among physicians, who tend to achieve higher diagnostic accuracy on skin types with which they are most familiar [ 62 ] . 1.2 Data Quality A key challenge in AI-based dermatological research is the heterogeneity of available datasets, given the diversity of skin diseases and imaging modalities. This study focuses on leveraging dermatoscopic images, which are typically more rare, for skin cancer identification. The most known datasets are HAM10000 [ 57 ] , ISIC [ 13 , 11 ] , and PH2 [ 41 ] for skin disease classification [ 35 , 7 ] . Additionally, recently introduced datasets, including crowdsourced repositories [ 61 ] and densely annotated datasets such as SkinCon [ 15 ] , further expand the resources available for model development but they don’t contain dermatoscopic images. While the increasing availability of dermatological images addresses the issue of data quantity, data quality is often neglected. This remains a critical oversight, as high-quality data is essential in machine learning, particularly for high-stakes applications like healthcare [ 34 ] . Considering the dermatoscopic images there are several factor that can impact the quality of the image: focus, angle, lightning, color representation [ 63 , 40 ] . Also image size plays