MetaLead : A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments
Roelien C. Timmer, Necva Bölücü, Stephen Wan · 2026
Read MetaLead : A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments by Roelien C. Timmer, Necva Bölücü, Stephen Wan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22420.txt.
Introduction
Roelien Timmer, Necva Bölücü, and Stephen Wan built MetaLead, a hand-checked data set for making leaderboards from papers. Leaderboards rank models and track progress in machine learning. Their 2026 preprint notes that past auto-built sets keep only each paper's best score and cover few fields.
Human curators built MetaLead to be broader and more exact. It can train and test tools that read papers and post their scores. Clear, full boards help the field trust its own progress claims. They also save hours of manual checking. The data set is the product here.
Catalog record
| CREATOR | Roelien C. Timmer, Necva Bölücü, Stephen Wan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22420
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MetaLead : A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments Roelien C. Timmer, Necva Bölücü, Stephen Wan Abstract Leaderboards are crucial in the machine learning (ML) domain for benchmarking and tracking progress. However, creating leaderboards traditionally demands significant manual effort. In recent years, efforts have been made to automate leaderboard generation, but existing datasets for this purpose are limited by capturing only the best results from each paper and limited metadata. We present MetaLead , a fully human-annotated ML leaderboard dataset that captures all experimental results for result transparency and contains extra metadata , such as the result experimental type (baseline, proposed method, or variation of proposed method) for experiment-type guided comparisons , and explicitly separates train and test dataset for cross-domain assessment . This enriched structure makes MetaLead a powerful resource for more transparent and nuanced evaluations across ML research. MetaLead dataset and code repository: https://github.com/RoelTim/metalead \DeclareCaptionType Text Box 1 Introduction The volume of scientific literature has grown exponentially 1 1 1 The scale of this increase is well demonstrated by arXiv, with a consistent annual growth in publication volume of approximately 10% 2 . , resulting in increased cognitive load for researchers, who struggle to compare existing work due to a scientific information overload 25 ; 4 . This information overload for researchers, combined with the fast-paced development of AI, has led to a growing supply of AI for Science tools designed to help researchers efficiently process and understand existing research, such as Google’s Co Scientist 12 , Ought’s Elicit 9 , and, OpenAI’s Deep Research 35 . For data science workflows, we note that the field already employs various conventions to assist in research planning and evaluation. For example, in Machine Learning (ML), a tool to easily compare the performance of different models is ML leaderboards . A popular online platform for ML leaderboards is Papers with Code 2 2 2 https://paperswithcode.com/ , which links research papers to their associated code implementations and maintains community-curated leaderboards across tasks. . However, manually curating these types of leaderboards is labour-intensive and requires continual manual maintenance. To ease this burden and enable cognitive offloading 3 3 3 Cognitive offloading refers to the use of external tools or strategies to reduce mental effort. 38 , researchers have developed NLP-based tools that automate the curation process 41 ; 13 ; 22 ; 19 ; 49 ; 20 . With the first publication in 2019, based on a BERT-style transformer encoder trained for natural language inference (NLI) 13 , while recent publications are based on multi-agent LLM frameworks 39 . Figure 1: MetaLead is more comprehensive than existing leaderboard datasets, with broader metadata (horizontal) and full inclusion of experimental results (vertical). Dataset All Results Annotation Source Scope Tuples 13 ✗ NLP Progress \langle task, dataset, metric, score \rangle 22 ✗ Papers with Code \langle task, dataset, metric, score \rangle 20 ✗ Papers with Code \langle task, dataset, metric, score \rangle 42 ✗ Papers with Code \langle task, dataset, metric, method, score \rangle 39 ✗ Human Annotation \langle task, dataset, metric, score \rangle MetaLead (Ours) ✓ Human Annotation \langle task, train dataset, test dataset, metric, score, experiment type \rangle Table 1: Comparison of MetaLead and related datasets. MetaLead uniquely includes all reported results, uses human annotation, and captures extended tuples with dataset splits and experiment type . Even though NLP methods for extracting ML leaderboard entries have advanced, the scope of leaderboards remains limited 46 . Existing leaderboards ( § 2 ) report only the best results, omitting ablations, negative results, and baselines. For instance, a recent shared task extracted only \langle task, dataset, metric, score \rangle tuples 8 , reflecting the field’s current focus while neglecting broader experimental context. The limited scope of current leaderboards leads to two key shortcomings . First, focusing only on best-performing results causes performance myopia 4 4 4 Performance myopia is the tendency to focus solely on best-performing results, neglecting broader context such as negative results, baselines, or failed variants. 10 and promotes publication bias, favouring positive results 23 . Second, the lack of distinction between train and test datasets hinders meaningful in-domain (ID) versus out-of-domain (OOD) comparisons, a critical distinction, as performance often varies across domains, leading to different conclusions and downstream consequences 45 ; 11 . To address these shortcomings, we introduce MetaLead , building on the previously annotated document set SciLead introduced by 39 , a manually annotated benchmark of scientific leaderboards that captures (task, dataset, metric, score) tuples from ML papers. Our additional annotations are illustrated in Figure 1 , adding 3,285 new records to the original 295 best records (i.e., leaderboard result) of 39 . Although MetaLead diverges from conventional leaderboards by including all reported results, it preserves the leaderboard’s core purpose: enabling comparative evaluation of model performance. This broader inclusion enhances transparency rather than undermining the concept. The three main contributions of this work are as follows: First, we introduce MetaLead , the first leaderboard-style dataset to include all experimental results from papers, enabling more transparency ( § 3.1 ). Second, we expand the schema of leaderboard tuples in two ways: (1) by introducing an explicit experiment type field, which enables experiment-type guided comparisons ( § 3.2 ) and is accompanied by a novel metric for evaluation ( § 5.3 ); and (2) by adding an explicit train/test dataset distinction , which supports cross-domain assessment ( § 3.3 ). Third, we provide a fully human-annotated resource ( § 4 ) and benchmarking results across various LLMs ( § 5 - § 6 ). Overall, MetaLead represents a step towards more holistic ML evaluation, broadening the notion of leaderboards and enabling more rigorous and transparent scientific inquiry. 2 Related Work ML Leaderboard Datasets MetaLead builds on prior work in ML leaderboard generation. Table 1 summarises existing datasets and shows how MetaLead compares. The comparison includes only datasets that report experimental results, as leaderboards are only meaningful when scores are provided. 5 5 5 E.g., 19 report \langle task, dataset, metric \rangle tuples but do not include scores. A key distinction of MetaLead is its comprehensive coverage: MetaLead includes all experimental results, whereas prior datasets report only the best results. Most existing datasets rely on community annotations, for example Papers with Code and its precursor NLP Progress 6 6 6 https://nlpprogress.com/ , a GitHub community-maintained NLP leaderboard. In contrast, only MetaLead and 39 provide fully human-annotated leaderboards, ensuring high-quality annotations. Tuple scope further differentiates datasets: most extract \langle task, dataset, metric, score \rangle tuples, while 42 reports \langle method, score \rangle conditioned on \langle task, dataset, metric \rangle . MetaLead ’s structure is the most detailed: \langle task, train dataset, test dataset, metric, score, experiment type \rangle , distinguishing training and test datasets and categorising results as baselines, proposed methods, or variations. Scientific Information Extraction Automatic ML leaderboard generation is part of a wider body of work about scientific information extraction (IE). 17 introduced SciREX, a benchmark dataset for scientific relation extraction focusing on identifying experimental results and their context within scientific documents. Similarly, 14 proposed TDM-Sci, which focuses on structured extraction of \langle task, dataset, metric \rangle triples from scientific papers. Other examples are IE used for scientific knowledge