We Need to Measure Data Diversity in NLP — Better and Broader
Dong Nguyen, Esther Ploeger · 2025
Read We Need to Measure Data Diversity in NLP — Better and Broader by Dong Nguyen, Esther Ploeger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20264.txt.
Introduction
Dong Nguyen and Esther Ploeger argue that natural language processing needs better and broader ways to measure data diversity, in this 2025 opinion paper. Diversity in language data sets draws growing attention, yet the question of how to measure it has been little studied. The authors examine the conceptual and methodological traps in current measures. They argue that views from other fields are essential if the measures are to become finer-grained and more valid. The piece is a call to treat diversity itself as a measurement problem worth solving properly.
Catalog record
| CREATOR | Dong Nguyen, Esther Ploeger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
We Need to Measure Data Diversity in NLP — Better and Broader Dong Nguyen, Esther Ploeger Abstract Although diversity in NLP datasets has received growing attention, the question of how to measure it remains largely underexplored. This opinion paper examines the conceptual and methodological challenges of measuring data diversity and argues that interdisciplinary perspectives are essential for developing more fine-grained and valid measures. 1 Introduction Language models seem to exhibit remarkable natural language understanding and generation capabilities. Yet numerous studies have pointed towards serious flaws: they encode societal biases ( Gallegos et al. 2024 ; Hofmann et al. 2024 ) and are sensitive to spurious correlations ( Du et al. 2023 ) . Many of these weaknesses have been traced back to the training data ( Feng et al. 2023 ; McCoy et al. 2024 ) . Moreover, as the size of these training datasets continues to increase, improvements in task performance show diminishing returns ( Tirumala et al. 2023 ) . Data size is often prioritized at the expense of data quality ( Paullada et al. 2021 ) . However, there is an increasing effort into analyzing the quality and composition of NLP datasets ( Albalak et al. 2024 ; Elazar et al. 2024 ; van Noord et al. 2025 ) . One aspect is the diversity of datasets, i.e., data diversity , which plays a key role in learning and generalization in both machine and human learning ( Raviv et al. 2022 ) . Smaller, more diverse datasets have enabled competitive model performance despite their size ( Zhou et al. 2023 ; Schiller et al. 2024 ) . Data diversity is also important for the fairness of models ( Santy et al. 2023 ) ; for example, models trained on primarily standard language perform worse on dialectal data ( Blodgett et al. 2018 ) . For evaluation, limited diversity in test data can be a cause for poor performance estimates ( Freitag et al. 2020 ) and hinders the ability to distinguish between different systems ( Sugawara et al. 2022 ) . Unfortunately, the concept of “data diversity” is challenging to define and measure, and researchers are rarely explicit about what they mean with data diversity ( Zhao et al. 2024 ) . But without clear definitions and measurements, it is difficult to make progress. Previous papers that advocate for measuring data diversity have left important aspects unaddressed. Mitchell et al. 2023 emphasise the importance of measuring data characteristics, though diversity in NLP data was not their main focus. Zhao et al. 2024 highlight the need to measure data diversity; however, their discussion mostly focuses on considerations during dataset construction, rather than diversity measurement itself. Shaib et al. 2025 call for standardizing text diversity measurements; however, their work is mostly empirical and focuses on lexical diversity. Further, the interdisciplinary focus of these papers is limited. Yet, data diversity measurement has long been of central importance in a wide range of fields, including language learning and assessment ( Jarvis 2013 ) , corpus linguistics ( Kilgarriff 2001 ; Egbert et al. 2022 ) , sociology ( Steele et al. 2022 ) and ecology ( Simpson 1949 ) . Across these and other fields, scholars have also grappled with how to define and quantify diversity. Our position is that an interdisciplinary approach is essential for improving data diversity measurements in NLP. Contribution In this opinion paper, we provide a critical perspective on the fundamental challenge of measuring dataset diversity in NLP . We focus on language data and associated information (e.g., about the authors or annotators). We operationally define data diversity as the variability of the data across specific dimensions (e.g., semantic, syntactic, lexical, sociolinguistic, genre, language, etc.) . Cross-lingual diversity has received increased attention in NLP Joshi et al. 2020 ; Ploeger et al. 2024 , leading to standardized language diversity metrics ( Ponti et al. 2020 ; Samardzic et al. 2024 ) . Within-language diversity, however, remains underexplored. For this reason, we focus on measuring diversity in monolingual settings. We first motivate why data diversity should be measured (§ 2 ). Then, we structure our discussion around three core challenges: diversity dimensions (§ 3.1 ), their measurement (§ 3.2 ), and evaluation of measurement (§ 3.3 ). Throughout our paper we highlight connections with other fields. Rather than offering definitive answers, we hope to spark discussion and to encourage more rigorous and interdisciplinary approaches to measuring data diversity. We conclude with concrete recommendations for NLP research that uses or develops diversity measures (§ 4 ). 2 Why Measure Data Diversity? Although many papers make qualitative judgements about the diversity of datasets (e.g., Zhou et al. 2023 ; Röttger et al. 2022 ), the exact nature and extent of the differences in diversity can remain unclear. We take the view that data diversity should be measured — that is, quantitatively characterized — along different dimensions ( Lai et al. 2020 ; Mitchell et al. 2023 ; Zhao et al. 2024 ) . Measuring properties of datasets is essential for advancing NLP as a science In Kilgarriff 2001 , Kilgarriff 2001 wrote: “ In any science, one expects to find a useful account of how its central constructs are taxonomised and measured, and how the subspecies compare. But to date, corpus linguistics has measured corpora only in the most rudimentary ways, ways which provide no leverage on the different kinds of corpora there are. ” (p. 97). This critique is also relevant to NLP: Quantitative analysis of dataset-level properties has been underexplored, with exceptions of recent work like Lohr and Hahn 2023 and Ramponi et al. 2024 . Among these properties, diversity has been highlighted in the literature ( Lai et al. 2020 ; Mitchell et al. 2023 ) . Measuring data diversity has the potential to improve every stage of the NLP pipeline First, data collection, especially when requiring human annotation, can be resource-intensive. Recent work ( Quattoni and Carreras 2021 ; Su et al. 2022 ; Shi et al. 2021 ; Alcoforado et al. 2024 ) shows that human annotation can be employed more effectively when selecting diverse datapoints. Second, including quantitative metrics on dataset diversity could further improve the documentation of datasets ( McMillan-Major et al. 2024 ; Gebru et al. 2021 ) and contribute to dataset analysis tools ( Lohr and Hahn 2023 ; Ramponi et al. 2024 ) . This could support early intervention, during data collection, instead of post hoc. For example, researchers could proactively collect more data from certain language varieties to improve the fairness of models. Third, leveraging data diversity measurements for training and prompting language models has led to better performing models, exhibiting improved robustness ( Bukharin et al. 2024 ) and generalization ( Levy et al. 2023 ) . Lastly, when it comes to evaluation, insufficiently substantiated diversity claims have caused disagreement on the generalizability of results. For example, an evaluation dataset that covers “a diverse range of text genres” ( Wang et al. 2018 ) may, by other standards, deemed “hardly diverse” ( Raji et al. 2021 ) . Systematic data diversity measurement could help researchers to assess and interpret the generalizability of evaluation results. Measuring data diversity can