Update (October 2025): This version includes a Corrigendum after the Appendix, correcting reported results for ViM and NNGuide. Please refer to the corrigendum section for details and updated tables. OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detection

Max Gutbrod, David Rauber, Danilo Weber Nunes, Christoph Palm · 2025

Read Update (October 2025): This version includes a Corrigendum after the Appendix, correcting reported results for ViM and NNGuide. Please refer to the corrigendum section for details and updated tables. OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detection by Max Gutbrod, David Rauber, Danilo Weber Nunes, Christoph Palm (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16247.txt.

Introduction

Max Gutbrod, David Rauber, Danilo Weber Nunes, and Christoph Palm build a benchmark suite for a safety problem in medical AI in this 2025 paper. Models trained on scans from one setting can fail when the input shifts to something unlike their training data. The suite tests how well methods can flag out-of-distribution inputs before they cause harm.

The suite, called OpenMIBOOD, holds three benchmarks from fourteen datasets across medical domains. Cases are sorted into shifted but valid inputs, near outliers, and far outliers. The team ran twenty-four detection methods through the same tests. A key lesson was that rankings from photo-based benchmarks did not carry over to medical images. The authors offer the suite as a yardstick so that detection methods for health care can be compared. This version of the paper also carries a corrigendum fixing reported results for two methods.

Catalog record

CREATORMax Gutbrod, David Rauber, Danilo Weber Nunes, Christoph Palm
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE124 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Update (October 2025): This version includes a Corrigendum after the Appendix, correcting reported results for ViM and NNGuide. Please refer to the corrigendum section for details and updated tables. OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detection Max Gutbrod, David Rauber, Danilo Weber Nunes, Christoph Palm Abstract The growing reliance on Artificial Intelligence in critical domains such as healthcare demands robust mechanisms to ensure the trustworthiness of these systems, especially when faced with unexpected or anomalous inputs. This paper introduces the Open Medical Imaging Benchmarks for Out-Of-Distribution Detection, a comprehensive framework for evaluating out-of-distribution detection methods specifically in medical imaging contexts. Open Medical Imaging Benchmarks for Out-Of-Distribution Detection includes three benchmarks from diverse medical domains, encompassing 14 datasets divided into covariate-shifted in-distribution, near-OOD, and far-OOD categories. We evaluate 24 post-hoc methods across these benchmarks, providing a standardized reference to advance the development and fair comparison of out-of-distribution detection methods. Results reveal that findings from broad-scale out-of-distribution benchmarks in natural image domains do not translate to medical applications, underscoring the critical need for such benchmarks in the medical field. By mitigating the risk of exposing Artificial Intelligence models to inputs outside their training distribution, Open Medical Imaging Benchmarks for Out-Of-Distribution Detection aims to support the advancement of reliable and trustworthy AI systems in healthcare. The repository is available at https://github.com/remic-othr/OpenMIBOOD . ID cs-ID near-OOD far-OOD MIDOG Scanner Cell Types + Scanner + Species Application PhaKIR Medium/Heavy Smoke Instruments + Organs + Species Application OASIS-3 Modality Scanner Anomalies Larger Anomalies Image Acquisition Application Figure 1 : Illustration of all utilized datasets, categorized by varying degrees of domain shift, from cs-ID to near-OOD and far-OOD, presented from left to right: MIDOG : All distinct MIDOG domains [ 5 ] , CCAgT [ 2 , 4 ] , FNAC 2019 [ 55 ] . PhaKIR : PhaKIR frames with smoke [ 53 ] , Cholec80 [ 66 ] , EndoSeg15 [ 10 ] , EndoSeg18 [ 1 ] , Kvasir-SEG [ 31 , 50 ] , CATARACTS [ 20 ] . OASIS-3 : T2w modality and distinct scanner from OASIS-3 [ 38 ] , ATLAS [ 43 ] , BraTS [ 47 ] , CT from OASIS-3 [ 38 ] , MSD-H [ 64 ] , CHAOS [ 32 ] . On top of each group the primary domain shift is presented. A ‘+’ indicates that the previous domain shift is also included. 1 Introduction The rapid proliferation of Artificial Intelligence systems in everyday life raises concerns about their trustworthiness. According to the technical specification ISO/IEC TS5723:2022 [ 30 ] , an important aspect of trustworthiness concerns a system’s ability to handle unexpected inputs. Such inputs pose significant risks in safety-critical areas, including autonomous driving [ 11 ] and the healthcare sector [ 74 ] . Most AI systems are still trained and evaluated under the assumption that the training data distribution – referred to as in-distribution data – matches the distribution of data encountered post-deployment. However, this assumption overlooks the possibility of encountering data from unknown and unseen distributions, known as out-of-distribution (OOD) data. When confronted with such data, Artificial Intelligence models often exhibit high confidence in their predictions, even when these predictions are entirely incorrect [ 18 ] . Such behavior can result in silent and potentially catastrophic failures, particularly in high-stakes domains like healthcare, where erroneous predictions could directly impact patient safety. To address this, out-of-distribution detection methods help distinguish in-distribution from out-of-distribution inputs, allowing models to flag or discard unreliable predictions or refer them for human review. Since 2016, numerous out-of-distribution detection methods have emerged [ 72 ] , but a unified, comprehensive benchmark was lacking. Yang et al . addressed this by introducing OpenOOD [ 72 ] , an open-source codebase that integrates related fields and provides a consistent evaluation framework. In their framework, out-of-distribution is divided into two categories. The more challenging near-OOD, characterized by exhibiting similar semantics or styles compared to the in-distribution datasets, making it particularly difficult to distinguish them. And the less challenging far-OOD, typically featuring different semantic labels and styles [ 16 , 72 ] . In a follow-up version, OpenOOD v1.5 [ 77 ] , Zhang et al . expanded their framework to encompass large-scale and full-spectrum out-of-distribution detection [ 73 ] , covering over 50 methods. In this full-spectrum setting, the authors merged in-distribution with covariate-shifted in-distribution (cs-ID) data – data, retaining the same labels and class relationships as the in-distribution data while exhibiting variations in the distribution of input features. They argue that this approach evaluates not only out-of-distribution detection capabilities but also the generalization performance of an Artificial Intelligence system. While this approach has undeniable value, we argue for analyzing out-of-distribution detection and model generalization independently to yield more precise insights. Furthermore, depending on the application, distinguishing between in-distribution and covariate-shifted in-distribution can be just as critical. For instance, in medical imaging, where training data is often limited or inaccessible due to privacy constraints, achieving robust generalization may be challenging. In such cases, focusing on detecting covariate-shifted in-distribution data can prove more impactful than solely aiming to enhance generalization performance. We propose that, rather than combining in-distribution and covariate-shifted in-distribution data, out-of-distribution detection methods should be rigorously evaluated based on their capacity to differentiate between these two categories. This is particularly important in domains like healthcare, where the ability to identify subtle covariate shifts in data is essential to maintaining reliability. To address the absence of medical images in the OpenOOD benchmark, we introduce three Medical Imaging Benchmarks, which consist of 14 datasets organized into three distinct out-of-distribution evaluation settings named after their in-distribution datasets: MIDOG [ 5 ] , PhaKIR [ 53 ] , and OASIS-3 [ 38 ] . An overview of these benchmarks and their involved domain shifts is provided in Fig. 1 . out-of-distribution detection methods are typically divided into two main types: post-hoc methods applied solely during inference, and methods requiring additional training steps. This work primarily focuses on post-hoc methods, valued for their ease of integration and model-agnostic nature. Besides these advantages, the analysis of Yang et al . showed that post-hoc methods are no worse than methods that require additional training steps [ 72 ] . Following an overview of the current benchmarks for out-of-distribution detection methods in the medical field ( Sec. 2 ), we describe all evaluation settings, their underlying datasets, and the employed evaluation metrics ( Sec. 3 ). Subsequently, we present the results of all evaluated methods, together with an in-depth discussion of these results ( Sec. 4 ) and a summary of key findings and future research directions ( Sec. 5 ). Our main contributions are as follows: • We introduce the most comprehensive out-of-distribution benchmarks for medical imaging across histopathology, endoscopy, and Magnetic Resonance Imaging domains, evaluating 24 post-hoc methods and revealing gaps in current approaches for certain domain shifts in medical data. • We show that selecting top-performing methods from benchmarks based on natural images likely leads to suboptimal performance on medical imaging data. • To reproduce our results, we provide a comprehensive codebase including all evaluated out-of-distribution detection methods, along with download links for the employed machine learning models and links to all utilized datasets.

Read the complete text