Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI’s Real World Effects
Reva Schwartz, Rumman Chowdhury, Akash Kundu, Heather Frase, Marzieh Fadaee, Tom David, Gabriella Waters, Afaf Taik, Morgan Briggs, Patrick Hall, Shomik Jain, Kyra Yee, Spencer Thomas, Sundeep Bhandari, Paul Duncan, Andrew Thompson, Maya Carlyle, Qinghua Lu, Matthew Holmes, Theodora Skeadas · 2025
Read Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI’s Real World Effects by Reva Schwartz, Rumman Chowdhury, Akash Kundu, Heather Frase, Marzieh Fadaee, Tom David, Gabriella Waters, Afaf Taik, Morgan Briggs, Patrick Hall, Shomik Jain, Kyra Yee, Spencer Thomas, Sundeep Bhandari, Paul Duncan, Andrew Thompson, Maya Carlyle, Qinghua Lu, Matthew Holmes, Theodora Skeadas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18893.txt.
Introduction
Benchmarks can say whether an AI system answers correctly, but not what changes when millions of people actually use it. A nineteen-author position paper argues that the field needs a new evaluation ecosystem aimed at second-order effects. Those include shifts in behavior, workforces, culture, and long-run outcomes in sectors like education, finance, health, and hiring.
Today's tests are mostly static, single-turn, and run in a lab, so they miss context entirely. The authors call for data and methods that follow AI into real settings, track what materializes over time, and support decisions about the fallout. They set out requirements for that ecosystem rather than building it. The paper's claim is that without such measurement, policy and design will keep flying blind about the effects that matter most.
Catalog record
| CREATOR | Reva Schwartz, Rumman Chowdhury, Akash Kundu, Heather Frase, Marzieh Fadaee, Tom David, Gabriella Waters, Afaf Taik, Morgan Briggs, Patrick Hall, Shomik Jain, Kyra Yee, Spencer Thomas, Sundeep Bhandari, Paul Duncan, Andrew Thompson, Maya Carlyle, Qinghua Lu, Matthew Holmes, Theodora Skeadas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18893
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI’s Real World Effects Reva Schwartz, Rumman Chowdhury, Akash Kundu, Heather Frase, Marzieh Fadaee, Tom David, Gabriella Waters, Afaf Taik, Morgan Briggs, Patrick Hall, Shomik Jain, Kyra Yee, Spencer Thomas, Sundeep Bhandari, Paul Duncan, Andrew Thompson, Maya Carlyle, Qinghua Lu, Matthew Holmes, Theodora Skeadas Abstract Conventional AI evaluation approaches concentrated within the AI stack exhibit systemic limitations for exploring, navigating and resolving the human and societal factors that play out in real world deployment such as in education, finance, healthcare, and employment sectors. AI capability evaluations can capture detail about first-order effects, such as whether immediate system outputs are accurate, or contain toxic, biased or stereotypical content, but AI’s second-order effects, i.e. any long-term outcomes and consequences that may result from AI use in the real world, have become a significant area of interest as the technology becomes embedded in our daily lives. These secondary effects can include shifts in user behavior, societal, cultural and economic ramifications, workforce transformations, and long-term downstream impacts that may result from a broad and growing set of risks. This position paper argues that measuring the indirect and secondary effects of AI will require expansion beyond static, single-turn approaches conducted in silico to include testing paradigms that can capture what actually materializes when people use AI technology in context. Specifically, we describe the need for data and methods that can facilitate contextual awareness and enable downstream interpretation and decision making about AI’s secondary effects, and recommend requirements for a new ecosystem. 1 Introduction As AI technologies have become mainstream, the number of tools for evaluating them have exploded within a highly active and competitive area of development and research. Measurement provides AI practitioners with the opportunity to test and learn whether and how the technology they build works once deployed 1 1 1 Measurement: (1) Quantitative measurement is the act or process of assigning a number or category to an entity to describe an attribute of that entity. ISO/IEC 24765:2017 (2) Qualitative measurement is based on descriptive data such as through observations, interviews, focus groups, or open-ended text fields in surveys. Evaluation enables interpretation of measurement results to place them into context. Metrology, the science of measurement, provides the methods and definitions of measurement that enable the evaluation of all measurement results, including for AI systems. Metrology provides the foundations for estimating measurement uncertainty that can incorporate multiple sources of random and systematic error. AI testing and evaluation is currently conducted within a computational and machine learning (ML) frame, with few systematic methods to account for the complex human, organizational and societal factors that inter-relate with the design, development, deployment and use of these technologies. This socio-technical 2 2 2 The term ”socio-technical systems” was coined in 1951 by Eric Trist and Ken Bamforth [ 101 ] to describe the dynamic ways workers interact with technological systems in industrial settings. framing of AI technology is currently difficult for ML practitioners to operationalize, or to know where, when and how to include which types of contextual information across the technology lifecycle. This paper argues that a new AI evaluation ecosystem is necessary to address current methodological gaps which impede the translation and contextualization of evaluation data and outcomes in the real world [ 107 , 104 , 16 , 27 , 29 , 33 , 85 , 82 , 35 , 96 , 38 , 80 , 76 ] . A real world AI evaluation ecosystem can enhance understanding of AI’s second-order effects, drive the collection of datasets that are fit-for-purpose, foster innovation, and improve AI functionality. 1.1 The Measurement Challenge The speed at which AI technology is advancing and being deployed and used across the globe [ 102 ] is not being met with equivalent evaluation paradigms for understanding its role and effects in societies. As a central topic of public policy efforts around the globe, questions about AI’s secondary effects abound. Private industry, civil society, the public, and governments around the world are increasingly interested in how AI technologies will transform our culture, economy, workforce and the broader society [ 69 , 75 , 73 , 74 , 41 , 44 ] . The current ecosystem to investigate these topics is fragmented, with no single evaluation toolbox or measurement infrastructure to account for AI’s second order effects and place them into the broader context. The predominant evaluation toolbox used by the ML research community, AI benchmarking, is designed to answer first order questions– about what AI systems can do based on direct measures of immediate system output. Another broad set of domains study AI’s human, organizational and societal factors, which tend to focus more on second order questions such as the effects associated with how people leverage AI technology, and how and why those effects reverberate across society. Other fields can place these findings into context to forecast future technological and societal trends. Some approaches, like user simulations, can simultaneously model AI user behavior and evaluate system performance. The AI metrology community is also deeply engaged in the development of tools to assess systems in more realistic settings with the broader goal of ensuring AI system trustworthiness. This work includes development of definitions [ 10 ] , and methods for calibration [ 106 ] , and uncertainty quantification [ 40 , 103 ] and propagation [ 100 ] . Yet, more effort is required, including efficient scalability and interpretation of measurement values. Table 1: Mapping evaluation approaches to effects measured and typical questions they answer. . Evaluation Approach Type of Effects (order) What it measures Answers questions like Benchmarking 1st Performance of the model/system in silico . 1. How often can the AI system produce the most accurate or relevant answer? 2. What is the inference runtime? 3. Did the model produce human-aligned responses? Testing & Evaluation 1st, 2nd Performance of the model/system in silico in vitro and in situ . 1. Does text summarization provide value for users? 2. Given current performance and user needs, should we expect productivity gains if we deploy this technology? If so, where? Verification & Validation 1st, 2nd Performance of the model/system in silico , in vitro and in situ .. 1. Does the AI system consistently generate video content per user specifications? 2. Does the AI system classify output according to vendor claims? Program Evaluation 2nd, 3rd Real-world efficacy and relevance in vitro and in situ . 1. Do AI assistants improve the quality of work? 2. How will AI-driven productivity gains transform different employment categories over time? Table 1 lists differences between the kinds of questions that can be answered by benchmarking, testing and evaluation, verification and validation, and program evaluation respectively. As methods move from first to second order and contextual detail increases, broader claims about AI technology become possible 3 3 3 In the context of AI evaluation, 1st-order effects are immediate system outputs, 2nd-order effects are longer-term impacts that may follow from system deployment, 3rd-order effects refer to broader changes that may result from AI’s role in society. In silico refers to testing conducted via computational methods. In situ refers to observing a phenomenon in its natural location or context. In vitro refers