A Position Paper on the Automatic Generation of Machine Learning Leaderboards

Roelien C. Timmer, Stephen Wan, Yufang Hou · 2025

Read A Position Paper on the Automatic Generation of Machine Learning Leaderboards by Roelien C. Timmer, Stephen Wan, Yufang Hou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17465.txt.

Introduction

Roelien Timmer, Yufang Hou, and Stephen Wan wrote a position paper in two thousand twenty-five on building machine learning leaderboards automatically. Leaderboards are tables that compare past experiments run under the same conditions. As the literature grows, keeping those tables complete and current by hand is getting harder.

Some researchers mine papers to fill leaderboard rows automatically. The authors give the first overview of that work. They show that studies frame the task in very different ways, which makes them hard to compare. They propose one shared framework for the task, plus guidelines for data sets and metrics so results can be judged fairly.

Catalog record

CREATORRoelien C. Timmer, Stephen Wan, Yufang Hou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17465

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Position Paper on the Automatic Generation of Machine Learning Leaderboards Roelien C. Timmer, Stephen Wan, Yufang Hou Abstract An important task in machine learning (ML) research is comparing prior work, which is often performed via ML leaderboards : a tabular overview of experiments with comparable conditions (e.g., same task, dataset, and metric). However, the growing volume of literature creates challenges in creating and maintaining these leaderboards. To ease this burden, researchers have developed methods to extract leaderboard entries from research papers for automated leaderboard curation. Yet, prior work varies in problem framing, complicating comparisons and limiting real-world applicability. In this position paper, we present the first overview of Automatic Leaderboard Generation (ALG) research , identifying fundamental differences in assumptions, scope, and output formats. We propose an ALG unified conceptual framework to standardise how the ALG task is defined. We offer ALG benchmarking guidelines , including recommendations for datasets and metrics that promote fair, reproducible evaluation. Lastly, we outline challenges and new directions for ALG , such as, advocating for broader coverage by including all reported results and richer metadata. 1 Introduction In today’s fast-paced Machine Learning (ML) research environment, keeping abreast of advancements is more crucial than ever. The exponential growth in publications, exemplified by nearly a quarter of a million arXiv submissions in 2024, underscores the expanding global community of scholars and the accelerating pace of research arXiv 2025 . This vast increase in information presents researchers with both rich opportunities for discovery but also makes it increasingly difficult to stay up to date. A key task for researchers is comparing past study outcomes to identify state-of-the-art results or benchmark against prior work. In ML, this is typically done using leaderboards: tables of experimental results under comparable conditions (e.g., task, dataset, metric ). The popularity of platforms like Papers with Code 1 1 1 https://paperswithcode.com underscores their value in providing accessible, up-to-date comparisons that help researchers track progress and identify leading methods. Figure 1: An example of extracting \langle task, dataset, metric, score \rangle tuples from research papers to build a leaderboard 3 3 3 An example of two SciLead Şahinüç et al. 2024 leaderboard entries summarising Lewis et al. 2020 and Dong et al. 2019 . . However, leaderboards on these platforms are often incomplete or missing for certain tasks, and they typically rely on manual updates. To reduce this manual effort, recent work has focused on automatically extracting experimental outcomes (referred to here as “tuples”) from research papers to populate leaderboards. We refer to this body of work as Automatic Leaderboard Generation (ALG) : “A systematic process for extracting relevant experimental findings from scientific publications to create and maintain a leaderboard.” . Figure 3 illustrates an example of this process, showing the extraction of \langle task, dataset, metric, score \rangle tuples from two research papers to construct a leaderboard. Research on ALG using natural language processing (NLP) methodologies has seen significant developments in recent years. Indeed, there are still many open research questions as exemplified by the 2024 shared task on ALG D’Souza et al. 2024 , underscoring the ongoing relevance of ALG. This growing body of work has led to varied problem formulations and evaluation approaches, including differing assumptions about prior knowledge ( § 2.1 ) and extraction scope ( § 2.2 ), which makes comparisons across work difficult. This position paper makes four important contributions. First, we provide the first overview of ALG efforts ( § 2 - § 4 ). By comparing prior studies side-by-side, we identify key divergences, such as variations in the assumed input scope (e.g., open vs. closed-domain) and captured results information, that previously hindered apples-to-apples comparisons. Our analysis provides a much-needed baseline map of the field, clarifying the field’s current state and identifying critical gaps. Second, based on this comparison, we propose an ALG unified conceptual framework ( § 5 ), essentially a problem formulation with unified terminology. This framework consolidates prior formulations into a coherent schema, providing a common language for researchers and enabling direct comparison of approaches. Third, we provide ALG benchmarking guidelines ( § 6 ), to unify evaluation practices, addressing the previous lack of consensus. These guidelines establish shared standards for consistent, transparent evaluation and reliable progress tracking. Fourth, we outline challenges and new directions for ALG ( § 7 ). We advocate expanding the extraction schema beyond just “best scores” to include all reported results (e.g., baselines, ablations) and enriching tuples with metadata (e.g., model architecture, hyperparameters) to enable more flexible result filtering. Ultimately, the goal of this position paper is to resolve long-standing fragmentation, establish shared standards, and open new horizons for ALG. 2 Overview of Problem Definition Methodology Domain Structured Data Scope of Extraction TDMS-IE Hou et al. 2019 closed Y \langle task, dataset, metric \rangle & best score PI Graph Singh et al. 2019 open Y undefined AxCell Kardas et al. 2020 closed Y \langle task, dataset, metric \rangle & best score SciREX-IE Jain et al. 2020 open Y \langle task, dataset, metric, method \rangle , no score ORKG-TDM Kabongo et al. 2021 closed Y \langle task, dataset, metric \rangle , no score TELIN Yang et al. 2022 open Y \langle task, dataset, metric \rangle , best score * ORKG-LB Kabongo et al. 2023b closed Y \langle task, dataset, metric \rangle , no score TDMS-PR Kabongo et al. 2024 open Y \langle task, dataset, metric \rangle & best score MS-PR Singh et al. 2024 open N \langle task \rangle & best score TDMR-PR Şahinüç et al. 2024 open N \langle task, dataset, metric \rangle & best score * The scope of extraction is ambiguous Yang et al. 2022 . A response from the authors is pending for clarification. Table 1: Characterisation of problem framing per method. Domain: open if extraction does not rely on prior knowledge, closed if restricted to a defined scope. Structured Data: Y if leaderboard tuples must appear in specific paper sections (e.g. tables or results), N otherwise. Scope of Extraction: extent of tuples extracted. The ALG field has seen many advances over the years. At a broad level, the ALG task is an information extraction task, to extract a tuple containing key details of an ML experimental result. 4 4 4 We acknowledge that ALG work rests on a long history of work in information extraction (IE) in scientific literature. The full body of IE work is out of scope for this analysis but is introduced briefly in Appendix A . Hou et al. 2019 and Singh et al. 2019 laid the foundation by introducing methods for extracting leaderboard tuples directly from research papers. These methodologies have since been refined and expanded upon by new methods such as AxCell Kardas et al. 2020 , which was put into production by Papers with Code. The most recent methodologies use prompting of pre-trained Large Language Models (LLMs), e.g. prompting Llama 2 7B Touvron et al. 2023 and Mistral 7B Jiang et al. 2023 to extract \langle task, dataset, metric, score \rangle tuples from research papers Kabongo et al. 2024 . A key issue in the field is the variation in input and output expectations across studies. Table 1 lists key ALG papers we examined, focusing on recent work using

Read the complete text