More than Marketing? On the Information Value of AI Benchmarks for Practitioners
Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M. Asmar, Sanmi Koyejo, Michael S. Bernstein, Mykel J. Kochenderfer · 2024
Read More than Marketing? On the Information Value of AI Benchmarks for Practitioners by Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M. Asmar, Sanmi Koyejo, Michael S. Bernstein, Mykel J. Kochenderfer (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05520.txt.
Introduction
Amelia Hardy and eight co-authors study how practitioners use AI benchmarks, in a paper from two thousand twenty-four. Public benchmark scores are widely advertised as signs of model quality. The authors asked whether those scores help the people who must choose and apply models in real work.
They interviewed nineteen people who had used benchmarks, or chosen not to, in daily work. Users treated scores as a signal of relative performance, not as final proof of superiority. Academics found public benchmarks fair measures of research progress, while product and policy users often found them inadequate. The authors argue good benchmarks need real-world tasks, domain expertise, and clear scope. They also need diverse skills, hard tests, and clean data.
Catalog record
| CREATOR | Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M. Asmar, Sanmi Koyejo, Michael S. Bernstein, Mykel J. Kochenderfer |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05520
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
More than Marketing? On the Information Value of AI Benchmarks for Practitioners Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M. Asmar, Sanmi Koyejo, Michael S. Bernstein, Mykel J. Kochenderfer Abstract . Public AI benchmark results are widely broadcast by model developers as indicators of model quality within a growing and competitive market. However, these advertised scores do not necessarily reflect the traits of interest to those who will ultimately apply AI models. In this paper, we seek to understand if and how AI benchmarks are used to inform decision-making. Based on the analyses of interviews with 19 individuals who have used, or decided against using, benchmarks in their day-to-day work, we find that across these settings, participants use benchmarks as a signal of relative performance difference between models. However, whether this signal was considered a definitive sign of model superiority, sufficient for downstream decisions, varied. In academia, public benchmarks were generally viewed as suitable measures for capturing research progress. By contrast, in both product and policy, benchmarks – even those developed internally for specific tasks – were often found to be inadequate for informing substantive decisions. Of the benchmarks deemed unsatisfactory, respondents reported that their goals were neither well-defined nor reflective of real-world use. Based on the study results, we conclude that effective benchmarks should provide meaningful, real-world evaluations, incorporate domain expertise, and maintain transparency in scope and goals. They must capture diverse, task-relevant capabilities, be challenging enough to avoid quick saturation, and account for trade-offs in model performance rather than relying on a single score. Additionally, proprietary data collection and contamination prevention are critical for producing reliable and actionable results. By adhering to these criteria, benchmarks can move beyond mere marketing tricks into robust evaluative frameworks that accurately reflect AI progress and guide informed decision-making in both research and practical domains. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY CCS: General and reference Evaluation CCS: Human-centered computing User studies CCS: Information systems Decision support systems CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Machine learning 1. Introduction When choosing between different AI models, users are faced with a growing number of increasingly capable options ( Maslej et al., 2024 ) . Such decisions may be informed by model evaluations, which commonly include manual assessment, red-teaming, and the use of AI benchmarks Chen et al. (2021) . Benchmarks, which following Raji et al. (2021) we define “as a particular combination of a dataset or sets of datasets […], and a metric, conceptualized as representing one or more specific tasks or sets of abilities, picked up by a community of researchers as a shared framework for the comparison of methods”, are frequently cited in publications and reports from model developer ( Tascilar, 2023 ; Bender et al., 2021 ) . Although such reports frame scores on public benchmarks as the basis one should have for understanding a model’s capabilities, it is not evident that this framing is consistent with how benchmarks are used in practice. Despite what developers who publish their scores may suggest, benchmark results may not give model users enough information to make substantive, final choices about which model is adequate for their use case. In this work, we seek to understand how AI benchmarks inform decisions between models. To build our theory, we perform and analyze semi-structured interviews with 19 practitioners in academia, policy, and industry, including research scientists, PhD students, and machine learning engineers. The study aims to address the following core research questions: • How are AI benchmarks developed, assessed, and used, across different application areas? • What challenges do stakeholders face in using and interpreting these benchmarks? • To what extent do AI benchmarks impact decision-making and progress in research, product, and policy? Our analysis reveals that in the domains of research, product, and policy, individuals are comfortable using benchmarks to make relative comparisons between models. These comparisons more frequently serve negative decisions more than positive ones: a low benchmark score can stop a model from being deployed, but a high score does not guarantee deployment. When it comes to real-world usage decisions, product and policy practitioners often find existing benchmarks to be insufficient. The ability of existing benchmarks to facilitate comparison does not obviate these practitioners’ need for further assessment, leading them to either develop internal benchmarks or to use alternate forms of evaluation. This contrasts with how benchmarks function in research settings, where the mere existence of a benchmark can drive growth within a sub-field. Thus, the question remains: what do benchmarks measure progress towards? We propose that the openness of this question is a key factor impacting the adoption of AI benchmarks. Drawing from theories of information technology acceptance ( Davis, 1989 ) , we find that although benchmarks meet the majority of the criteria for technology adoption, they fall short on the factor of performance expectancy, i.e., a technology’s perceived ability to perform the task it is assigned. A recurring theme among participants in our study was that of a gap between the task a benchmark tests a model on and how that model will ultimately be used. While there are fundamental limits to how well a test or simulation task can approximate real-world applications ( Kadian et al., 2020 ) , we found that participants were able to overcome part of the gap by developing internal evaluations to better reflect their actual use cases. We find that benchmark developers can make their benchmarks more informative and useful to practitioners by considering user personas and use cases during the design process, incorporating feedback from domain experts, and clearly documenting how results should and should not be interpreted. 2. Related Work Our literature review considers three broad domains as the basis for understanding how AI benchmarks are (and are not) used to support decisions. First, to position our findings, we review the literature on the recognized issues with current AI evaluation practices. Although some issues with model assessment are benchmark-specific, others remain open questions in the field. Second, to contextualize current AI benchmarks, we review literature on the history of benchmarking and analyze the motivations that have driven past benchmark use in computing and other fields. Finally, to map the circumstances under which benchmarks may be adopted, we consider models for the acceptance of information technology. 2.1. Evaluating AI Evaluating AI systems is necessary for understanding their capabilities, limitations, and potential for positive and negative impacts ( Saxon et al., 2024 ) . However, this process is complex, requiring careful consideration of which evaluation methods are used and how their results are interpreted. One of the primary challenges in AI evaluation is ensuring the closeness of an evaluation to a system’s anticipated use ( Raji et al., 2021 ; Beede et al., 2020 ) . This issue is exemplified by a study conducted by ( leb, 2021 ) , which examined five different AI tools deployed in a U.S. hospital. The study found that none of these tools met expectations created by accuracy measured using expert-labeled ground truth data. This highlights the risk of treating constructed quality measures as objective markers of knowledge and