What Does Your Benchmark Really Measure? A Framework for Robust Inference of AI Capabilities

Nathanael Jo, Ashia Wilson · 2025

Read What Does Your Benchmark Really Measure? A Framework for Robust Inference of AI Capabilities by Nathanael Jo, Ashia Wilson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19590.txt.

Introduction

Benchmark scores are treated as direct readings of what an AI model can do, and Nathanael Jo and Ashia Wilson say that habit hides a shaky step. A score becomes a claim about skill only through a theory of what skill at the task means. Fields such as psychometrics make that theory explicit. AI testing, the authors argue, mostly leaves it unsaid.

They frame evaluation as an inference task that should rest on a stated capability theory. As a proof of concept, they show that reported performance can shift strongly with the evaluator's modeling choices. Two teams can thus read the same test in unlike ways. The position paper calls for transparent, theory-driven testing. Clearer foundations would steady both public debate and research claims about new models.

Catalog record

CREATORNathanael Jo, Ashia Wilson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19590

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Does Your Benchmark Really Measure? A Framework for Robust Inference of AI Capabilities Nathanael Jo, Ashia Wilson Abstract Evaluations of generative models on benchmark data are now ubiquitous, and their outcomes critically shape public and scientific expectations of AI’s capabilities. Yet growing skepticism surrounds their reliability. How can we know that a reported accuracy genuinely reflects a model’s true performance? Evaluations are often presented as simple measurements, but in reality they are inferences: to treat benchmark scores as evidence of capability is already to assume a theory of what capability is and how it manifests in a test. We make this step explicit by proposing a principled framework for evaluation as inference: begin from a theory of capability, and then derive methods for estimating it. This perspective, familiar in fields such as psychometrics, has not yet become commonplace in AI evaluation. As a proof of concept, we address a central challenge that undermines reliability: sensitivity to perturbations. After formulating a model of ability, we introduce methods that infer ability while accounting for uncertainty from sensitivity and finite samples, including an adaptive algorithm that significantly reduces sample complexity. Together, these contributions lay the groundwork for more reliable and trustworthy estimates of AI capabilities as measured through benchmarks. Part 1: Start with a theory of AI capability Part 2: Develop inference strategies from that theory Figure 1: Diagram of our proposed framework. 1 Introduction Evaluations (from hereon, “evals”) of generative models have become ubiquitous as a way to probe each models’ capabilities or harms. Companies developing large language models (LLMs) routinely assess their systems’ intelligence using standardized knowledge tasks, while research papers proposing new methods often conduct comparative evaluations against state-of-the-art models. Leaderboards hosted on Vellum and Huggingface have also emerged as open-source platforms for directly comparing the capabilities of various LLMs. The rapidly growing interest in evals reflects our collective desire to understand how generative models behave, especially as they are now widely utilized, touted as highly capable, but inherently black box in nature. In this work, we focus on evals that use standardized benchmark datasets. Despite the many criticisms of benchmarking ( Alzahrani et al. 2024 ; Raji et al. 2021 ) , benchmark data still comprise the vast majority of evals today. Benchmark results can carry significant influence in calibrating public and scientific expectations of AI capabilities 1 1 1 Notably, companies that develop LLMs publish heavily cited reports on their models’ performance on various benchmarks ( Achiam et al. 2023 ; Team et al. 2023 ; Guo et al. 2025 ) . , which motivates the need for robust and trustworthy methods for inferring and reporting these capabilities. Yet, there is growing consensus that generative AI evals using benchmarks are brittle and unreliable ( Mitchell 2023 ; Eriksson et al. 2025 ) . We believe two key reasons contribute to this lack of reliability. The first is that most evals report point estimates without quantifying uncertainty . This practice is common even among the most widely cited evaluations ( Hendrycks et al. 2021 ; Lin et al. 2022 ; Achiam et al. 2023 ; Srivastava et al. 2023 ; Liang et al. 2023 ) , which offer little interpretation of the significance of their results (e.g., in comparing two models’ capabilities on a given task). This stands in stark contrast to evaluations of classical predictive models, which were historically grounded in statistical inference and hypothesis testing ( Rainio et al. 2024 ) . Recent efforts such as Chatbot Arena 2 2 2 Though admittedly not based on benchmark data. ( Chiang et al. 2024 ) and Miller 2024 have begun to advocate for calculating and reporting confidence intervals. However, these works remain focused primarily on finite-sample uncertainty. The second issue is that evals are rarely grounded in sound theories of AI capability . Most report standard metrics (e.g., accuracy) without considering what it actually means for an AI system to be “capable” of a task. High benchmark accuracy is, at best, a weak proxy for capability. For example, a growing body of work shows that minor changes to the phrasing or structure of inputs can significantly alter model outputs ( Mizrahi et al. 2024 ; Sclar et al. 2023 ; Zhuo et al. 2024 ; Zheng et al. 2023 ; Errica et al. 2024 ) . Yet this type of uncertainty — how models respond to the natural variation in user queries — is rarely captured in benchmark evals. We argue that evals in the era of generative models must not only return back to its roots of statistical inference, but also carefully define what “capabilities” truly mean. To that end, we draw inspiration from psychometrics, where statistical models have long been used to robustly estimate latent traits of human ability. Our work aims to develop a theory of AI capability that incorporates various sources of uncertainty in service of informing robust methods for inferring generative models’ capabilities. Contributions. (1) We propose a principled framework to robustly infer AI capabilities from benchmark data: start from a theory of performance, and develop methods for inference from that theory . As a proof of concept, we tackle a crucial problem that confounds evaluations: model sensitivity to perturbations . Our proposed theories of capability are intuitive and clearly expose why traditional benchmarks might produce biased estimates due to violated statistical assumptions. (2) We empirically demonstrate that existing LLM benchmarks indeed violate these assumptions, leading to systematically inflated or deflated performance estimates. This finding aligns with previous works that have similarly quantified AI sensitivity to perturbations ( Mizrahi et al. 2024 ; Sclar et al. 2023 ) , but we are the first to quantify this phenomenon in the context of systematic bias in measuring performance. We also show that even state-of-the-art models (e.g., gpt-4.1 ) suffer from this bias, and discuss implications of evaluating larger, more capable models. (3) Finally, building on our theory of capabilities, we propose two inference techniques: (a) clustered bootstrapping to estimate accuracy ; and (b) an adaptive test based on item response theory to estimate a model’s latent ability . The latter method imposes some additional assumptions but significantly reduces the number of evaluations required and can better identify a more robust notion of capability. 1.1 Related Work Psychometric principles in AI evals. Our model and methods are largely inspired from psychometric testing. Wang et al. 2023 provide a comprehensive overview of the many ways psychometrics can provide a useful lens for improving AI evaluations. Notably, a good test must be (1) valid , meaning that it properly measures the construct of interest; and (2) reliable , in that the resulting measure is replicable and consistent. Improving construct validity is indeed an important open problem in AI benchmarks ( Raji et al. 2021 ) , but our work focuses on improving reliability by proposing methods that provide credible intervals on performance. Critiques of AI evals. Many works have proposed ways to improve AI assessments through benchmarking. For example, Siska et al. 2024 challenge the representativeness of certain questions in a benchmark on real-life tasks; Myrzakhan et al. 2024 convert multiple choice benchmarks to open-ended questions and show that LLMs perform worse on open-ended questions. Both

Read the complete text