Benchmarked Yet Not Measured - Generative AI Should be Evaluated Against Real-World Utility

Ishani Mondal, Shweta Bhardwaj · 2026

Read Benchmarked Yet Not Measured - Generative AI Should be Evaluated Against Real-World Utility by Ishani Mondal, Shweta Bhardwaj (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06856.txt.

Introduction

Ishani Mondal and Shweta Bhardwaj argue that generative AI should be judged by real-world utility, not benchmark scores. Across twenty-eight deployment cases in education, healthcare, software engineering, and law, they find systems that shine on benchmarks yet fail users. They name three recurring evaluation failures: proxies replacing real goals, snapshots replacing change over time, and averages hiding uneven effects across groups.

Their proposed SCU-GenEval framework runs in four stages, from mapping stakeholder goals to measuring utility longitudinally as capability changes through sustained use. Supporting tools include structured deployment protocols, context-conditioned user simulators, and persona-aware proxy metrics. The paper closes with domain-specific calls to action for outcome-based evaluation.

Catalog record

CREATORIshani Mondal, Shweta Bhardwaj
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06856

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarked Yet Not Measured - Generative AI Should be Evaluated Against Real-World Utility Ishani Mondal, Shweta Bhardwaj Abstract Generative AI systems achieve impressive performance on standard benchmarks yet fail to deliver real-world utility—a disconnect we found across 28 deployment cases spanning education, healthcare, software engineering, and law. We argue that this benchmark–utility gap arises from three recurring failures: proxy displacement , temporal collapse and distributional concealment , reinforced by Goodhart-style optimization on saturated benchmarks and a broader failure of construct validity in current evaluation practice. Motivated by these, we argue that generative AI evaluation requires a paradigm shift: from static benchmark-centered transparency toward stakeholders, goals, and context-conditioned utility transparency grounded in human outcome trajectories. Existing evaluations primarily characterize properties of model outputs, while deployment success depends on whether interaction with AI improves stakeholders’ ability to achieve their goals over time. The missing construct is therefore utility : the change in a stakeholder’s capability induced through sustained interaction with an AI system within a deployment context. To operationalize this perspective, we propose SCU-GenEval , a four-stage evaluation framework consisting of stakeholder–goal mapping, construct–indicator specification, mechanism modeling, and longitudinal utility measurement. To make these stages practically deployable, we introduce three supporting instruments: structured deployment protocols, context-conditioned user simulators, and persona–goal-conditioned proxy metrics. We conclude with domain-specific calls to action, arguing that progress in generative AI must be evaluated by measurable improvements in human outcomes rather than by benchmark performance alone. 1 Introduction Benchmarks such as ImageNet ( 67 ) , GLUE ( 75 ) , HumanEval ( 13 ) , and VBench ( 32 ) have been central to the progress of generative AI by enabling standardized comparison across models. However, as generative AI systems move into education, healthcare, software engineering, and other high-stakes settings, a growing disconnect has emerged between benchmark performance and real-world usefulness. Current evaluation pipelines largely focus on artifact-level metrics such as BLEU ( 57 ) , ROUGE ( 43 ) , pass@k ( 13 ) , fluency, preference win-rates, and visual realism. In deployment, however, the critical question is fundamentally different: does interaction with the system improve human capability, decision-making, learning, collaboration, and trust over time in real environments? To investigate this benchmark–utility disconnect, we analyze 28 documented deployment cases spanning education, healthcare, and software engineering (Figure 1 ; detailed in Table 1 ). Across domains, we observe a recurring pattern: systems that perform strongly on standard benchmarks often fail to deliver reliable real-world utility. In software engineering, code-generation systems with high pass@k scores still introduce security vulnerabilities in nearly 40% of generated solutions ( 59 ) . In healthcare, the widely deployed Epic Sepsis model missed 67% of sepsis cases while incorrectly alerting on 18% of patients despite strong internal validation ( 78 ; 28 ) . In education, systems that improve short-term task completion frequently fail to improve long-term retention or transfer, while AI-assisted programming tools can reduce metacognitive engagement and contribute to skill atrophy over time. Collectively, these findings suggest that benchmark success is often a weak proxy for the human outcomes deployment seeks to improve. Failure Mode Case IDs and Description Proxy displacement S6 (fabricated citations, fluency), S10 (Copilot \sim 40% vulnerabilities), S13 (coverage overstates fault-detection), S15 (BLEU misses doc errors), S16 (cost \neq health need), S22 (medical LLM hallucinations), S25 (COMPAS AUC \neq contestable justice), S26/S28 (fluent legal citations \neq valid authority) Temporal collapse S1 (ITS transfer unmeasured), S3 (ChatGPT completion vs. transfer), S4 (AI summaries skip retrieval), S5 (CS1 Copilot metacognitive harm), S14 (developer skill decay), S24 (chatbot short-term-only gains) Distributional concealment S7 (low-performers gain less), S16 (Black patients under-enrolled), S17 (COVID classifier confounders), S18 (GPT-4 racial/gender bias), S20 (Watson site-generalization failure), S27 (SafeRent score obscured voucher/race disparities) Figure 1: Benchmark–utility divergence in generative AI evaluation. The figure illustrates three recurring structural failures of current evaluation: proxy displacement , temporal collapse , and distributional concealment . The table below preserves full set of representative cases for each failure. From these cases, we identify three recurring structural failures underlying the benchmark–utility gap. First, proxy displacement occurs when tractable benchmark metrics are treated as valid measures of complex real-world constructs without evidence of construct validity. Metrics such as BLEU, ROUGE, pass@k, and preference win-rates are routinely interpreted as indicators of semantic fidelity, code quality, or usefulness despite weak alignment with deployment outcomes ( 68 ) . This also manifests when evaluations focus only on outcomes while ignoring whether systems follow acceptable reasoning processes: transparency, contestability, or domain-specific procedural values specially in high-stakes domains ( 44 ) . Second, temporal collapse occurs when short-term or AI-assisted performance is mistaken for durable capability. Existing evaluations frequently measure immediate task success while ignoring retention, transfer, over-reliance, metacognitive decline, calibration, and long-term skill development 63 ; 77 . Third, distributional concealment manifests when average scores mask the failures across populations, under-represented groups, or unseen deployment settings, allowing systems that appear robust on average to systematically fail for minority populations, novice users, or unseen environments ( 81 ) . Importantly, these failures are not isolated. They reinforce one another through benchmark saturation, contamination, memorization, and Goodhart-style optimization dynamics ( 27 ; 3 ; 36 ; 31 ; 10 ) . More fundamentally, they expose a broader problem of construct validity ( 15 ; 34 ) : current benchmarks are rarely validated against the deployment outcomes they are implicitly assumed to represent ( 51 ) . Prior work has already questioned whether automatic metrics meaningfully capture human utility ( 69 ; 65 ; 79 ) , while research in human–AI interaction demonstrates that usefulness varies substantially across users, goals, and contexts ( 26 ) . Together, these findings suggest that aggregate benchmark scores provide only partial transparency about deployed AI systems. Position. These observations motivate the central position of this paper: generative AI evaluation requires a paradigm shift from static benchmark-centered transparency toward stakeholder-conditioned, goal-conditioned, and task-conditioned utility transparency centered on human outcome trajectories and capability shifts in specific deployment context . Existing benchmarks primarily provide transparency about model outputs ; deployment instead requires transparency about how interaction with AI changes human capability over time . Under this view, evaluation should not merely report whether a model produces high-quality outputs in isolation, but instead characterize whose capabilities change, under which goals and deployment contexts, through which mechanisms, and across what temporal horizons. In other words, the unit of evaluation — and therefore the unit of transparency — must shift from artifact quality to the human outcome trajectory reflecting their change in capability . Our Proposal. Based on these observations, we argue that the missing concept in current evaluation is utility ( 20 ; 45 ) : not simply the quality of an output artifact, but the extent to which interaction with an AI system changes a person’s ability to achieve their goals over time. To address this gap, we propose Stakeholder-Centered Utility Evaluation (SCU-GenEval) , a framework for evaluating generative AI systems through deployment-grounded human outcomes. The framework operationalizes utility through four connected stages: identifying stakeholders and deployment goals, defining constructs and measurable indicators, modeling the mechanisms through which AI changes user behavior and capability, and finally measuring utility longitudinally and across subgroups. In this sense, SCU-GenEval is not

Read the complete text