FunnyNodules: A Customizable Medical Dataset Tailored for Evaluating Explainable AI

Luisa Gallée, Yiheng Xiong, Meinrad Beer, Michael Götz · 2025

Read FunnyNodules: A Customizable Medical Dataset Tailored for Evaluating Explainable AI by Luisa Gallée, Yiheng Xiong, Meinrad Beer, Michael Götz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15481.txt.

Introduction

Luisa Gallée, Yiheng Xiong, Meinrad Beer, and Michael Götz introduce FunnyNodules, a synthetic medical image data set built to test whether AI models reason for the right reasons. Real lung scans rarely come with labels for the visual traits behind a diagnosis, which makes such checks hard. FunnyNodules instead generates abstract nodule-like shapes with traits that can be set by hand.

Traits such as roundness, edge sharpness, and spiky margins combine through a fixed rule to decide each case class, so the true reasoning is fully known. Users can vary difficulty, class balance, and target rules. Tests show how to check whether a model learns the right trait links and looks at the right image regions.

Catalog record

CREATORLuisa Gallée, Yiheng Xiong, Meinrad Beer, Michael Götz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15481

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FunnyNodules: A Customizable Medical Dataset Tailored for Evaluating Explainable AI Luisa Gallée, Yiheng Xiong, Meinrad Beer, Michael Götz Abstract Densely annotated medical image datasets that capture not only diagnostic labels but also the underlying reasoning behind these diagnoses are scarce. Such reasoning-related annotations are essential for developing and evaluating explainable AI (xAI) models that reason similarly to radiologists: making correct predictions for the right reasons. To address this gap, we introduce FunnyNodules , a fully parameterized synthetic dataset designed for systematic analysis of attribute-based reasoning in medical AI models. The dataset generates abstract, lung nodule–like shapes with controllable visual attributes such as roundness, margin sharpness, and spiculation. Target class is derived from a predefined attribute combination, allowing full control over the decision rule that links attributes to the diagnostic class. We demonstrate how FunnyNodules can be used in model-agnostic evaluations to assess whether models learn correct attribute–target relations, to interpret over- or underperformance in attribute prediction, and to analyze attention alignment with attribute-specific regions of interest. The framework is fully customizable, supporting variations in dataset complexity, target definitions, class balance, and beyond. With complete ground truth information, FunnyNodules provides a versatile foundation for developing, benchmarking, and conducting in-depth analyses of explainable AI methods in medical image analysis. \midlauthor 1,2 \orcid 0000-0001-5556-7395 1 Experimental Radiology, Ulm University Medical Center, Ulm, Germany 2 XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology, Ulm, Germany and 1,2 \orcid 0009-0009-9847-6034 and 2,3 \orcid 0000-0001-7523-1979 3 Department of Diagnostic and Interventional Radiology, Ulm University Medical Center, Ulm, Germany and 1,2 \orcid 0000-0003-0984-224X † † volume: – Under Review † † year: 2026 † † workshop: Full Paper – MIDL 2026 submission † † editors: Under Review for MIDL 2026 keywords Dataset, Explainable AI, Evaluation 1 Introduction In medical image analysis, numerous machine learning models and explainable AI (xAI) methods have been proposed 32 ; 29 ; 8 ; 3 ; 23 . While model performance is typically well evaluated, other aspects, such as the correctness of model reasoning, that is, whether a model makes the right decision for the right reasons, are often insufficiently assessed 28 . Even for xAI methods that are explicitly designed to improve interpretability, systematic evaluation remains a major challenge 23 . A key reason for this limitation is the lack of comprehensive ground truth information. In addition to annotations for the target classes, the evaluation of xAI methods also requires ground truth for the visual explanations at the sample level. Such annotations are rare, particularly in the medical domain where dataset sizes are inherently limited 22 . To address this gap, we introduce FunnyNodules, a synthetic image dataset specifically designed for evaluating AI and xAI methods in medical imaging. A core feature of FunnyNodules is its comprehensive annotation. Since the image generation process is explicitly controlled, all samples follow predefined appearance features (referred to as attributes), similar to the approach of Hesse et al. 17 , who applied this concept to natural images with discrete object parts (e.g., bird beak or wing) as features, and therefore did not include medical-specific aspects (e.g., lesion intensity or shape). The parameterized generative approach enables the automatic creation of complete ground truth information, including target class and attribute labels as well as region-of-interest (ROI) masks, without being affected by inter- or intra-rater variability. Furthermore, the FunnyNodules framework provides a high degree of customization. Both the variability in image appearance and the decision rules for target classification can be adapted to different levels of complexity. This enables researchers to tailor the dataset to the specific evaluation needs of their models. In contrast to existing synthetic datasets generated using large generative models such as Diffusion Models 26 ; 19 ; 15 or GAN 9 , the goal of FunnyNodules is not to simulate realistic data. Instead, it focuses on simulating attribute relationships with clearly defined ground truth and full controllability. In addition, this approach does not require any real training data and is free from data-driven biases. FunnyNodules provides a controlled environment for evaluating AI models and xAI methods in medical image analysis, allowing the investigation of model reasoning behavior without the limitations of real-world datasets. In this paper, we • introduce FunnyNodules, a synthetic vision dataset inspired by medical image interpretation and designed for systematic evaluation of AI models as explainability methods. • demonstrate how FunnyNodules can be used to assess model behavior with respect to attribute sensitivity, reasoning correctness, and trustworthiness. The dataset generation code and all presented experiments are publicly available at https://github.com/XRad-Ulm/FunnyNodules. 2 Dataset The aim of this synthetic dataset is to enable comprehensive evaluation of AI models for medical images. Full control over the image generation process allows the creation of depictions that follow defined visual attributes. This approach offers high customizability, scalability, and targeted adjustments, providing strong flexibility for model-agnostic evaluations. The FunnyNodules framework is based on the idea of representing abstract lung nodules through controllable visual attributes. Rating the malignancy of lung nodules is a clinically relevant and well-studied task in radiology 10 ; 7 . The diagnostic evaluation strongly depends on quantifiable visual features, such as intensity, roundness, and margin sharpness 2 ; 1 , which directly correspond to the concept of attributes in explainable AI. 2.1 FunnyNodules The FunnyNodules dataset, employed in the experiments of this study, comprises abstract nodules described by six visual attributes: \bullet roundness (r) 1- round , 5- oval \bullet spiculation (sp) 1- none , 5- marked \bullet edge sharpness (es) 1- sharp , 5- soft \bullet size (s) 1- small , 5- big \bullet intensity (i) 1- dark , 5- bright \bullet internal structure (is) 0- absent , 1- present The target class is defined based on combinations of these attributes (see Algorithm 2 ) and is ordinal, ranging from 1 to 5, the same as all attributes except internal structure, which is binary. However, a major advantage of the synthetic FunnyNodules framework is the ability to implement different scales and rules as desired. For performance evaluation, we use the Within-1-Accuracy metric, following prior attribute-based studies 21 ; 11 . Predictions are considered correct if they deviate by at most \pm 1 from the ground truth score for ordinal labels, whereas internal structure requires exact agreement. target : 1 r: 3 sp: 1 es: 1 s: 1 i: 1 is: 1 target : 2 r: 3 sp: 4 es: 4 s: 3 i: 3 is: 0 target : 4 r: 1 sp: 1 es: 1 s: 5 i: 3 is: 1 target : 5 r: 4 sp: 4 es: 4 s: 4 i: 1 is: 0 target : 4 r: 3 sp: 5 es: 4 s: 5 i: 2 is: 0 target : 4 r: 2 sp: 1 es: 2 s: 4 i: 5 is: 1 {algorithm2e} [H] Table 1: target score\leftarrow 0 \If is=0 \lIf r\geq 4 score\leftarrow+2 \lElseIf r\leq 2 score\leftarrow-2 \ElseIf is=1 \lIf r\geq 4 score\leftarrow-2 \lElseIf r\leq 2 score\leftarrow+2 \lIf \quad\;\;\;\,\,sp\geq 4 score\leftarrow+2 \lElseIf sp\leq 2 score\leftarrow-2 \lIf es\geq 4 score\leftarrow-2 \lElseIf es\leq 2 score\leftarrow+2 \lIf \qquad\;\,\,s\geq 4 score\leftarrow+2 \lElseIf \;\,s\leq 2 score\leftarrow-2 \lIf \qquad\;\,\,\,i=5 score\leftarrow-1 \lElseIf \;\,\,i\leq 2 score\leftarrow+1 \lIf score\leq-1\;\;\, target\leftarrow 1 \lElseIf score=0\;\;\;\,\,\,\, target\leftarrow 2 \lElseIf score=[1,2] target\leftarrow 3 \lElseIf score=[3,4] target\leftarrow 4 \lElse \qquad\qquad\qquad\qquad\;\;\;\;\;\,\,target\leftarrow

Read the complete text