MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events

Raunak Agarwal, Markus Wenzel, Simon Baur · 2026

Read MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events by Raunak Agarwal, Markus Wenzel, Simon Baur (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15203.txt.

Introduction

Agarwal and colleagues launch MADE, a living benchmark for multi-label classification of medical-device adverse-event reports. Older benchmarks are saturating and may leak training data, blurring reasoning and memorization. MADE draws on continuously added published reports with strict time-based splits and a long-tailed hierarchy of labels. Baselines across over 20 models show clear trade-offs: small tuned decoders lead on accuracy, generative tuning gives the steadiest uncertainty, and self-stated confidence proves unreliable.

Catalog record

CREATORRaunak Agarwal, Markus Wenzel, Simon Baur
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64215
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15203

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events Raunak Agarwal, Markus Wenzel, Simon Baur Abstract Machine learning in high-stakes domains such as healthcare requires not only strong predictive performance but also reliable uncertainty quantification (UQ) to support human oversight. Multi-label text classification (MLTC) is a central task in this domain, yet remains challenging due to label imbalances, dependencies, and combinatorial complexity. Existing MLTC benchmarks are increasingly saturated and may be affected by training data contamination, making it difficult to distinguish genuine reasoning capabilities from memorization. We introduce MADE , a living MLTC benchmark derived from m edical device ad verse e vent reports and continuously updated with newly published reports to prevent contamination. MADE features a long-tailed distribution of hierarchical labels and enables reproducible evaluation with strict temporal splits. We establish baselines across more than 20 encoder- and decoder-only models under fine-tuning and few-shot settings (instruction-tuned/reasoning variants, local/API-accessible). We systematically assess entropy-/consistency-based and self-verbalized UQ methods. Results show clear trade-offs: smaller discriminatively fine-tuned decoders achieve the strongest head-to-tail accuracy while maintaining competitive UQ; generative fine-tuning delivers the most reliable UQ; large reasoning models improve performance on rare labels yet exhibit surprisingly weak UQ; and self-verbalized confidence is not a reliable proxy for uncertainty. Our work is publicly available at https://hhi.fraunhofer.de/aml-demonstrator/made-benchmark . 1 Introduction Figure 1: Top: Overview of the benchmarking setup, encompassing discriminative and generative language models, learning paradigms (discriminative or generative fine-tuning and few-shot prompting), and uncertainty quantification (UQ) approaches. Bottom, left: Multi-label text classification of medical device adverse events, each annotated with hierarchical product and patient problem labels. Bottom, right: UQ quality is evaluated (for of each model, learning paradigm and UQ method). Strong predictive performance is not sufficient for the adoption of machine learning (ML) models in high-stakes settings such as healthcare ( 31 ; 58 ) where human oversight is paramount ( 61 ) . In this regard, uncertainty quantification (UQ) is essential for reliable ML systems ( 51 ) because it can flag doubtful or ambiguous cases for human re-examination. Multi-label text classification (MLTC) is central to patient categorization, clinical coding, and incident reporting, etc. Developing reliable MLTC systems presents practitioners with several challenges: MLTC requires selecting multiple labels from a typically much larger set, which leads to a combinatorial problem that scales exponentially with the label space size. Real-world MLTC data can be characterized by severe inter- and intra-class imbalances: a few common conditions comprise the majority of examples, while safety-critical conditions reside in the long tail. Models must learn to disentangle correlated signatures without becoming biased toward frequent classes. Further, labels often co-occur and are hierarchically interdependent, violating the assumption of label independence. Crucially, existing MLTC benchmarks are increasingly ill-suited for evaluating frontier large language models (LLMs). Traditional datasets are static, can be saturated, suffer from data contamination due to their inclusion in LLM pre-training corpora, or lack the label imbalance and interdependence of real-world environments. These challenges leave practitioners with open questions: Which model architecture is most suitable for this complexity? Can a specialized, small model solve the task, or is a larger foundation model required? Which learning paradigm (fine-tuning vs. in-context learning) yields the best trade-off between performance for frequent and rare classes? How reliable are the resulting predictions? We address these questions by introducing an uncontaminated benchmark and systematically evaluating models across architectures, learning paradigms, and UQ methods. 1.1 Related work Multi-label text classification. Early approaches to MLTC relied on models like Naïve Bayes, support vector machines ( 69 ) , or recurrent neural networks. With the advent of transformer-based language models ( 68 ) , encoder-only ( 20 ) and decoder-only ( 40 ; 15 ) architectures served for this task. Controlled comparisons between generative and discriminative fine-tuning for decoder models and head-to-head comparisons between encoder- and decoder-only models under matched conditions are rare. 71 addresses this with a suite of matched encoder- and decoder-only models (‘Ettin’) trained on the ModernBERT architecture ( 70 ) across model sizes, but do not evaluate on MLTC, leaving this comparison open. Decoder-only LLMs support zero-shot (inference without labeled data) and few-shot prompting (in-context learning; conditioning on a few labeled exemplars without updating model weights), which has been compared with fine-tuning on a variety of tasks where medical expertise is required, with differing results ( 50 ; 41 ; 30 , e.g.,) . Across broader benchmarks, the advantage of fine-tuning vs. prompting varies by task, data regime, and model ( 8 ; 72 ) . For MLTC, performance is sensitive to e.g. label semantics, output constraints, and thresholding strategies, motivating controlled comparisons. Benchmarking datasets for multi-label text classification span diverse domains, including scientific literature ( 29 , 76 , 13 , 7 ; 59 ), newswire ( 32 ) , finance-related user posts ( 42 ) , patents ( 64 ) , clinical notes for ICD coding ( 23 ; 47 ) , legislative documents ( 63 , 5 , 6 , 4 ), and more. Overlap between benchmark content and LLM pre-training corpora raises concerns about contamination that can inflate zero-/few-shot performance 22 ; 56 ; 78 ; 33 ; 11 ; 74 . Heavy reuse can lead to benchmark saturation via overfitting and implicit adaptation, which motivates the continuous introduction of novel datasets to assess generalization. Typical benchmarks are scraped from online repositories and rely on distant or weak supervision (e.g., tags/metadata as labels), often with limited documentation, introducing label noise and hampering reproducibility. Multiple sources, pre-processing choices, copyright limitations, and dataset versions further complicate fair comparisons. Uncertainty quantification (UQ). For discriminative models, entropy-based UQ measures are well understood ( 60 ; 46 ; 1 ) . In contrast, UQ for generative models is more complex: token-level probabilities, consistency across stochastic generations, and self-verbalized confidence introduce distinct challenges. However, estimating uncertainty in LLM benchmarks is of great interest ( 2 ) and recent works ( 77 ; 66 , e.g.,) cover different natural language processing tasks, but not yet MLTC. Token- or information-based uncertainty metrics, which in various forms rely on the log-probabilities of an LLM’s outputs, have been surveyed and formalized by 62 ; 14 . Among those of interest are: entropy of the top- n log-probabilities, perplexity, average or maximum token log-probability. Consistency-based UQ metrics measure the output variability of a model under stochasticity, and have been widely used for LLMs as a way to capture epistemic uncertainty ( 73 ) . 36 suggested an effective approach to derive a consistency-based uncertainty score through the sum of eigenvalues of a graph Laplacian. 67 proposed the combination of information- and consistency-based metrics as an effective strategy to improve uncertainty estimation, motivating our approach of integrating token-level uncertainty with consistency measures. Self-verbalized uncertainty, in which a model outputs a confidence score corresponding to its prediction 24 ; 65 ; 18 , offers a complementary approach to directly capture model-reported uncertainty, especially when log-probabilities are not available. 12 review confidence calibration for imbalanced data. We focus on information-based and consistency-based metrics due to their complementary strengths, and additionally test self-verbalized uncertainty given its practical appeal when log-probabilities are unavailable. 1.2 Our contributions In this paper, we (1) create a benchmark: To counteract saturated benchmarks

Read the complete text