Dataset Diversity Metrics and Impact on Classification Models
Théo Sourget, Niclas Claßen, Jack Junchi Xu, Rob van der Goot, Veronika Cheplygina · 2026
Read Dataset Diversity Metrics and Impact on Classification Models by Théo Sourget, Niclas Claßen, Jack Junchi Xu, Rob van der Goot, Veronika Cheplygina (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15276.txt.
Introduction
Sourget and co-authors study how dataset diversity metrics behave and whether they predict classifier performance. Diversity is often praised but rarely defined the same way twice. They test metrics for images, text, and metadata on MorphoMNIST and the PadChest chest X-ray set. Reference-free image and metadata metrics linked weakly to AUC. FID and semantic diversity metrics linked much more strongly. A clinical expert named scanners as the main real-world diversity source, yet adding a scanner triggered shortcut learning in the model.
Catalog record
| CREATOR | Théo Sourget, Niclas Claßen, Jack Junchi Xu, Rob van der Goot, Veronika Cheplygina |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36085 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dataset Diversity Metrics and Impact on Classification Models Théo Sourget, Niclas Claßen, Jack Junchi Xu, Rob van der Goot, Veronika Cheplygina Abstract The diversity of training datasets is usually perceived as an important aspect to obtain a robust model. However, the definition of diversity is often not defined or differs across papers, and while some metrics exist, the quantification of this diversity is often overlooked when developing new algorithms. In this work, we study the behaviour of multiple dataset diversity metrics for image, text and metadata using MorphoMNIST, a toy dataset with controlled perturbations, and PadChest, a publicly available chest X-ray dataset. We evaluate whether these metrics correlate with each other but also with the intuition of a clinical expert. We also assess whether they correlate with downstream-task performance and how they impact the training dynamic of the models. We find limited correlations between the AUC and image or metadata reference-free diversity metrics, but higher correlations with the FID and the semantic diversity metrics. Finally, the clinical expert indicates that scanners are the main source of diversity in practice. However, we find that the addition of another scanner to the training set leads to shortcut learning. The code used in this study is available at https://github.com/TheoSourget/dataset_diversity_evaluation 1 Introduction Figure 1 : Overview of our study. We assessed dataset diversity measures across multiple modalities and compare their correlations as well as their alignment with domain experts knowledge through interviews. We evaluate their impact on downstream task performances and the effect on subgroups training dynamics. While diversity is often said to be an important dataset property to train accurate and robust models [ 14 , 21 ] , medical image analysis papers often focus on methodology to improve the downstream performances, overlooking the evaluation of the data used in the studies [ 40 ] . Moreover, the definition of diversity itself is not always clear and may vary across papers [ 45 ] . It can for example relate to the actual content of the data (e.g. the variety of words in text datasets or the visual aspects in image datasets), to the label distribution, or to meta-information such as the demographics of patients. While medical imaging datasets are often of limited size compared to natural imaging datasets, larger datasets are becoming available [ 3 , 23 , 29 ] . However, the assumption that larger datasets are necessarily more diverse and representative is not always true [ 41 ] . In the medical context, the addition of new data may even damage the performance of a classifier, especially in multi-center datasets where inclusion of data from other hospitals may introduce a distribution shift or a potential shortcut [ 7 , 37 ] . The increasing size of datasets and sometimes their curation from online sources [ 20 ] have also been shown to introduce quality problems [ 2 ] . Several metrics exist to measure the diversity of data, often used in the context of generative models to ensure that they produce diverse set of samples [ 18 , 26 , 35 ] . These metrics are yet to be evaluated on medical image classification datasets. Moreover, it is still unclear whether an improvement on these diversity metrics actually translates to better downstream performances or if they align with clinicians’ interpretation of diversity. Finally, most studies focus on evaluating diversity of a single modality, which is not sufficient for increasingly popular multimodal datasets [ 3 , 23 ] . Our contributions, aiming for a better understanding of diversity metrics and their impact on model trainings in various applied scenarios, are as follows: 1. We assess image, text, and metadata diversity metrics on a toy synthetic dataset and on chest X-ray data, providing insights into their strengths and weaknesses in different scenarios. 2. We evaluate their impact on models’ performance, fairness, robustness, and training dynamics, finding which metrics are suitable for the selection of a training set. 3. We conduct a semi-structured interview with a radiology resident to gather the clinical view of diversity and discuss its alignment with diversity metrics and research practices. 2 Related Work There are multiple definitions of dataset diversity, yet it is not always clear which definition is used when a paper addresses diversity. In their taxonomy, Zhao et al . [ 45 ] identify five definitions said to increase diversity: (1) the content of the data (e.g. variety of objects in images or words in texts), (2) the location of the data collection, (3) the domain of the data (e.g. type of images or texts), 4) the demographics within the dataset, and 5) the background of the annotators. Closely related to diversity, Clemmensen and Kjærsgaard [ 6 ] discuss how representativity of datasets is described in research papers. They similarly find that representativity is not always clearly defined, and sometimes simply claimed without supporting arguments. In our work, we choose to explicitly focus on the diversity in the content of the data and the demographics within the dataset. Other studies have focused on the measures for dataset quality. Mitchell et al . [ 31 ] detail metrics to quantify five different aspects of machine learning datasets: (1) distance, (2) density, (3) diversity, (4) tendency, and (5) association. They categorize the metrics as “General” (e.g. Gini diversity and Vendi Score) or “Modality-specific” (e.g. lexical diversity, Inception Score, and subset diversity). Dang et al . [ 9 ] also present a taxonomy for data quality and metrics for natural language processing (NLP) datasets, including diversity, defined as the “variety and range covered by the dataset”. They compare five metrics: the Shannon entropy, the Renyi entropy, the Simpsons index, the Vendi Score, and the inter-class distance introduced in their study. Diversity is considered an important characteristic of generative models outputs to match the entire training distribution and avoid the generation of identical samples (also referred to as mode collapse). Multiple works address its quantification [ 1 , 11 , 19 , 32 ] , Dombrowski et al . [ 11 ] also introduce a new metric, the Image Retrieval Score as the number of real images being the closest to at least one of the synthetic image. Closely related to our study, Mironov et al . [ 30 ] highlights the limitations of existing diversity measures using simple failures cases. They define three desirable properties for a diversity metric: monotonicity, uniqueness, and continuity. They categorize existing metrics according to whether they satisfy these properties and propose two metrics that satisfy all criteria, but which are not usable in practice due to their computational cost. Finally, Konz et al . [ 25 ] show the limitations of existing measures to compare two sets of medical images and present the Fréchet Radiomic Distance (FRD) using clinically meaningful features within images. They use the FRD to evaluate image-to-image translation models on a large range of medical downstream tasks, and show that the FRD is better correlated with the downstream task metric like the AUC or the Dice score. We extend these works by applying diversity metrics across different modalities (image, text, and tabular) in the context of medical classification datasets, providing insights into their behaviour, advantages and failure cases when applied to