Toward an evaluation science for generative AI systems

Laura Weidinger Google DeepMind Inioluwa Deborah Raji University of California, Berkeley Hanna Wallach Microsoft Research Margaret Mitchell Hugging Face Angelina Wang Stanford University Olawale Salaudeen Massachusetts Institute of Technology Rishi Bommasani Stanford University Deep Ganguli Anthropi · 2025

Read Toward an evaluation science for generative AI systems by Laura Weidinger Google DeepMind Inioluwa Deborah Raji University of California, Berkeley Hanna Wallach Microsoft Research Margaret Mitchell Hugging Face Angelina Wang Stanford University Olawale Salaudeen Massachusetts Institute of Technology Rishi Bommasani Stanford University Deep Ganguli Anthropi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05336.txt.

Introduction

Laura Weidinger and co-authors argue that testing generative AI systems needs to become a science of its own, in work from 2025. Such systems are now used in real settings, but common tests are fixed lists that may not predict real-world behaviour. One-off audits, they note, rarely scale to wide use.

The authors draw lessons from fields that learned to test safety the hard way, such as transport, aerospace, and drug making. Three lessons stand out. Tests must match real-world use, tests must be refined over time, and shared bodies and norms must run the testing. They then lay out a path toward stricter, more trusted checks. The piece is a call to treat AI testing as a mature field, not an afterthought.

Catalog record

CREATORLaura Weidinger Google DeepMind Inioluwa Deborah Raji University of California, Berkeley Hanna Wallach Microsoft Research Margaret Mitchell Hugging Face Angelina Wang Stanford University Olawale Salaudeen Massachusetts Institute of Technology Rishi Bommasani Stanford University Deep Ganguli Anthropi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05336

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward an evaluation science for generative AI systems Laura Weidinger Google DeepMind Inioluwa Deborah Raji University of California, Berkeley Hanna Wallach Microsoft Research Margaret Mitchell Hugging Face Angelina Wang Stanford University Olawale Salaudeen Massachusetts Institute of Technology Rishi Bommasani Stanford University Deep Ganguli Anthropic Sanmi Koyejo Stanford University William Isaac Google DeepMind Abstract There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts. However, the current evaluation ecosystem is insufficient: commonly used static benchmarks face validity challenges, and ad hoc case-by-case approaches rarely scale. In this piece, we advocate for maturing an evaluation science for generative AI systems. While generative AI creates unique challenges for system safety engineering and measurement science, the field can draw valuable insights from the development of safety evaluation practices in other fields including transportation, aerospace, and pharmaceutical engineering. In particular, we present three key lessons: evaluation metrics must be applicable to real-world performance, metrics must be iteratively refined, and evaluation institutions and norms must be established. Applying these insights, we outline a concrete path toward a more rigorous approach for evaluating generative AI systems. 1 Introduction The widespread deployment of generative AI systems in medicine ( Boyd 2023 ) , law (e.g., Lexis+AI 1 1 1 https://www.lexisnexis.com/en-us/products/lexis-plus-ai.page ), education ( Singer 2023b ) , information technology (e.g., Microsoft’s Copilot 2 2 2 copilot.microsoft.com , Reid 2024 , and many social settings (e.g. Replika 3 3 3 replika.com , character.AI) has led to a collective realization: the performance and safety of generative AI systems in real-world deployment contexts are very often poorly anticipated and understood ( Roose 2024a ; Mulligan ; Wiggers 2024 ) . The tendency of these systems to generate inaccurate statements has already led to the spreading of medical and other misinformation ( Omiye et al. 2023 ; Archer and Elliott 2025 ) ; incorrect legal references ( Magesh et al. 2024 ) ; failures as educational support tools ( Singer 2023a ) ; and widespread confusions in search engine use ( Heaven 2022 ; Murphy Kelly 2023 ) . Beyond factual discrepancies, AI-enabled chatbots have also been described as interacting inappropriately with users ( Roose 2023 ) , exposing security vulnerabilities ( Nicholson 2023 ) , and fostering unhealthy emotional reliance ( Roose 2024b ; Dzieza 2024 ; Verma 2023 ) . The historical focus on benchmarks and leaderboards has been effective at encouraging the AI research community to pursue shared directions; however, as AI products become widely integrated into our everyday lives, it is increasingly clear that static benchmarks are not well suited to improving our understanding of the real-world performance and safety of deployed generative AI systems ( Bowman and Dahl 2021 ; Goldfarb-Tarrant et al. 2021a ; Liao et al. 2021 ; Raji 2021 ; de Vries et al. 2020 ) . Despite this mismatch, static benchmarks are still commonly used to inform real-world decisions about generative AI systems that stretch far beyond the research landscape – such as in deployment criteria and marketing materials for new model or system releases ( Grattafiori et al. 2024 ; Anthropic 2024b ; OpenAI 2024 ; Gemini et al. 2024 ) , third party critiques ( Mirzadeh et al. 2024 ; Zhang et al. 2024 ) , procurement guidelines ( Johnson et al. 2025 ) , and in public policy discourse ( European Commission 2024 ; National Institute of Standards and Technology 2021 ) . Although there is an emerging interest in more interactive ( Chiang et al. 2024 ) , dynamic ( Kiela et al. 2021 ) , and behavioral ( Ribeiro et al. 2020 ) approaches to evaluation, many of the existing alternatives to benchmarks, such as red teaming exercises and case-by-case audits, still fall woefully short of enabling systematic assessments and accountability ( Birhane et al. 2024 ; Friedler et al. 2023 ) . For AI evaluation to mature into a proper “science,” it must meet certain criteria. Sciences are marked by having theories about their targets of study, which can be expressed as testable hypotheses. Measurement instruments to test these hypotheses must provide experimental consistency (i.e., reliability, internal validity) and generalizability (i.e., external validity). Finally, sciences are marked by iteration: Over time, measurement approaches and instruments are refined and new insights are uncovered. Collectively, these properties of sciences contrast sharply with the practice of rapidly developing static benchmarks for evaluating generative AI systems, while anticipating that within a few months such benchmarks will become much less useful or obsolete. As generative AI exits an era of research and enters a period of widespread use ( Hu 2023 ; Reid 2024 ) , the field risks exacerbating an ongoing public crisis of confidence in AI technology ( Faverio and Tyson 2023 ) if we do not develop a more mature evaluation science for generative AI systems. From the history of other fields, we can get a sense of why: Collectively, leaderboards, benchmarks, and audits do not amount to the robust and meaningful evaluation ecosystem we need to properly assess the suitability of these products in widespread use. In particular, they cannot give assurances about AI system performance in different domains or for different user groups 4 4 4 We thank our anonymous reviewer for pointing out that one reason for this is that benchmarks are often not robust to data or domain shifts, i.e. benchmarks test AI system outputs in certain contexts but this may not be predictive of AI system behaviour in other contexts. . In this piece, we advocate for the maturation of such an evaluation science. By drawing on insights from systems safety engineering and measurement science in other fields, while acknowledging the unique challenges inherent to generative AI, we identify three important properties of any evaluation science that the AI community will need to focus on to meaningfully advance progress: a focus on real-world applicability, iterative measurement refinement, and adequate institutional investment. These properties then enable us to outline a concrete path toward a more rigorous evaluation ecosystem for generative AI systems. 2 Lessons from Other Fields The bridges we stand on, the medicine we take, and the food we eat are all the result of rigorous assessment. In fact, it is because of the rigor of the corresponding evaluation ecosystems that we can trust that the products and critical infrastructure surrounding us are performant and safe. Generative AI products are no exception to this reality and therefore not unique in their need for robust evaluations. In response to their own crises, more established evaluation regimes emerged in other fields to assure users and regulators of safety and reliability - offering concrete lessons for the AI field ( Rismani et al. 2023 ; Raji and Dobbe 2023 ; Raji et al. 2022a ) . We note three key evaluation lessons from these other fields: the targeting of real-world performance, the iterative refinement of measurement approaches, and the establishment of functioning processes and institutions. 2.1 Real-world applicability of metrics First, it is noteworthy that, historically, evaluation made a difference for safety because it tracked real-world risks. Measuring real-world performance does not mean waiting until risks manifest —- on the contrary, earlier pre-deployment risk detection and

Read the complete text