Position: State-of-the-Art Claims Require State-of-the-Art Evidence

YongKyung Oh · 2026

Read Position: State-of-the-Art Claims Require State-of-the-Art Evidence by YongKyung Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17273.txt.

Introduction

Oh challenges state-of-the-art claims in AI, arguing that top average scores alone cannot carry them. A claim of superiority implies meaningful effect size, consistency across tasks and robustness to dropping datasets. Yet in over half of top-model comparisons across ten public benchmarks at least one of those traits fails. Aggregate wins often ride on outlier datasets, so claim language should mirror the strength of the evidence.

Catalog record

CREATORYongKyung Oh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63198
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Position: State-of-the-Art Claims Require State-of-the-Art Evidence YongKyung Oh Abstract State-of-the-Art (SOTA) claims pervade Artificial Intelligence (AI) and Machine Learning (ML) research. These claims rest on benchmark evaluations, where models are ranked by aggregate scores across tasks. Public benchmarks or leaderboards are the most visible instance, but the same structure appears in paper tables throughout the literature. However, such minimal evidence often cannot support these strong claims. We identify a widespread claim-evidence gap in AI benchmarking. Claiming SOTA carries implicit assumptions beyond mean score superiority, suggesting that a model meaningfully outperforms alternatives across most tasks. However, a marginal improvement in the mean score merely indicates a top average rank rather than true superiority. Analyzing ten cross-domain benchmarks from public leaderboards, we found that in more than half of top-model comparisons, at least one commonly assumed property of superiority does not hold. These properties include meaningful effect size, consistency across tasks, or robustness to dataset removal. Instead, aggregate gains are frequently driven by outlier datasets. This fragility persists even in benchmarks with many tasks. We argue that claim language should reflect the strength of the underlying evidence. This requires no additional experiments, only honest reporting of what results actually show, enabling more precise and interpretable comparisons across models. 1 Introduction State-of-the-Art (SOTA) claims shape how Artificial Intelligence (AI) and Machine Learning (ML) research measures progress. Models are compared across benchmark tasks, and the top performer by aggregate score is declared superior. Figure 1 illustrates the scale of the problem. The number of accepted papers at major AI conferences has grown rapidly. Throughout this expansion, a substantial proportion of papers, frequently exceeding 30% at major venues, explicitly claim state-of-the-art performance in their abstracts. Figure 1 : Published papers mentioning “state-of-the-art” in abstracts across major AI conferences (2021–2025). Darker bars indicate SOTA mentions, and lighter bars indicate other papers. In this study, the empirical analysis in Section 4 focuses on benchmarks that aggregate across multiple tasks or datasets. Please refer to Appendix A for details on the terminology, scope, and methodology used in Figure 1 . A mean score across k tasks establishes only that a model ranks first on average. It does not show that the model wins on most tasks or would retain its rank under different dataset compositions. 10 documented that statistical tests are routinely misapplied or omitted entirely. Prior work demonstrates that variance arising from data sampling, hyperparameter selection, and test set construction is often comparable to reported improvements ( 5 ) . However, marginal gains in aggregate scores continue to be interpreted as evidence of robust superiority. This claim-evidence gap is pervasive across modalities. In natural language processing, rankings are demonstrably sensitive to task composition and prompt formatting, often reversing leadership based on minor perturbations ( 9 ; 2 ; 30 ; 17 ) . Computer vision benchmarks exhibit similar volatility, where top-ranked models frequently degrade under distribution shifts ( 37 ; 34 ) . Analogous patterns plague time series forecasting, where selective dataset reporting allows a vast majority of methods to claim spurious superiority ( 41 ) . Collectively, these findings confirm that benchmark-based rankings are inherently fragile. Despite widespread awareness, claim language remains unchanged. Researchers continue to claim “SOTA” based on the same thin evidence that prior work has shown to be unreliable. Thus, this paper makes three contributions: • We identify a claim-evidence gap. A mean score establishes only that a model ranks first on average. It does not establish that the model wins consistently, improves meaningfully, or would retain its rank under different conditions. However, claiming “state-of-the-art” carries implicit assumptions beyond mean score superiority. Our diagnostics make these assumptions explicit and testable. • We show the gap is pervasive. We apply elementary statistical diagnostics to ten public leaderboards. We find that more than half of top-model comparisons lack support for at least one commonly assumed property of superiority. This pattern holds across benchmarks that aggregate semantically distinct tasks, separate datasets within a single task, or different evaluation dimensions. • We argue that closing the gap requires institutional norms. The diagnostics we apply require no new experiments. The persistence of the problem despite widespread awareness confirms that the barrier is institutional, not technical. Individual authors cannot shift equilibria alone, but venue-level guidelines can. Our analysis addresses benchmark-based SOTA claims, that is, comparisons where models are ranked by aggregate scores across tasks. This structure dominates major venues, though some work uses application-specific evaluation. We are not criticizing benchmark design. Public benchmarks have accelerated progress by enabling reproducible comparison. Our concern is with interpretation: claim language that exceeds what the evidence supports. We argue that claim language in the ML literature often exceeds what the computed evidence supports. Papers compute factual evidence but use language reserved for robust superiority. We support this argument by analyzing ten major benchmarks drawn from public leaderboards. The diagnostic methods apply to any benchmark comparison, whether published on a leaderboard or reported in a paper table. We call for venue-level norms that encourage claim-evidence alignment. This requires no new experiments, only honest interpretation of existing results. Conflict of Interest Disclosure. No financial conflicts. 2 Related Work 2.1 Statistical Comparison Methods Prior work has established statistical frameworks for model comparison centered on the Wilcoxon signed-rank test ( 52 ; 15 ; 53 ) , which remains a standard recommendation for pairwise evaluation. 10 showed that paired t-tests over cross-validation folds inflate Type I error, leading to a preference for non-parametric alternatives. In practice, critical-difference diagrams ( 10 ) are used in time-series studies ( 24 ; 32 ) . This analysis was extended to post-hoc comparisons and multiple testing corrections ( 16 ; 6 ; 11 ) . However, we address a distinct problem: even when appropriate tests are applied, claim language often exceeds what the evidence supports. Variance from hyperparameter selection often rivals reported improvements ( 5 ) . Similarly, cross-validation error estimation exhibits structural variance dominated by training set sensitivity ( 40 ) . Also, 4 showed that marginal p -values yield low replication probability. Nevertheless, adoption remains limited. Our approach aligns with this diagnostic perspective without necessitating experimental redesign. Instead, it focuses on the implementation of disciplined reporting. 2.2 Benchmark Fragility 9 introduced the “benchmark lottery” to describe how rankings depend on task selection. 41 quantified dataset selection bias in time series forecasting. Minor protocol changes such as prompt formatting can cause substantial ranking shifts ( 2 ; 45 ) . Human-preference evaluation shows similar sensitivity. Position and verbosity biases change rankings when not controlled ( 55 ) . Furthermore, 30 showed that performance varies substantially across semantically identical questions. 17 found systematic errors in MMLU that change model rankings when corrected. Similarly, 37 documented accuracy drops under distribution shift even on mature benchmarks. Rigorous reporting practices exist, though they remain rare. Chatbot Arena ( 7 ) reports Bradley-Terry coefficients with confidence intervals rather than point rankings. FEV-Bench ( 46 ) provides bootstrap confidence intervals on win rates, revealing that apparent gaps between top models are often not statistically distinguishable. These examples demonstrate that uncertainty quantification is feasible at negligible cost. Their rarity reflects institutional barriers rather than technical ones. While prior research has diagnosed benchmark fragility and proposed technical remedies

Read the complete text