Making AI Evaluation Deployment-Relevant Through Context Specification This work was conducted as part of the Forum for Real‑World AI Measurement and Evaluation (FRAME) at Virginia State University’s Center for Responsible AI

Matthew Holmes, Thiago Lacerda, Reva Schwartz · 2026

Read Making AI Evaluation Deployment-Relevant Through Context Specification This work was conducted as part of the Forum for Real‑World AI Measurement and Evaluation (FRAME) at Virginia State University’s Center for Responsible AI by Matthew Holmes, Thiago Lacerda, Reva Schwartz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06811.txt.

Introduction

Holmes and colleagues introduce context specification, a process for making AI evaluation useful for real deployment choices. Status quo evaluations often hide day-to-day realities that decide whether tools deliver lasting value, leaving leaders unsure how systems will behave in managed settings. It turns diffuse stakeholder views about what matters into clear named constructs. Those are clear definitions of traits, behaviors, and outcomes that evaluations target, so teams can observe and measure them in context. The process serves as a roadmap for judging likely deployment conduct.

Catalog record

CREATORMatthew Holmes, Thiago Lacerda, Reva Schwartz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34382
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06811

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Making AI Evaluation Deployment-Relevant Through Context Specification This work was conducted as part of the Forum for Real‑World AI Measurement and Evaluation (FRAME) at Virginia State University’s Center for Responsible AI Matthew Holmes, Thiago Lacerda, Reva Schwartz Abstract With many organizations struggling to gain value from AI deployments [ 19 ] , pressure to evaluate AI in an informed manner has intensified. Status quo AI evaluation approaches mask the operational realities that ultimately determine deployment success, making it difficult for decision‑makers outside the stack to know whether and how AI tools will deliver durable value. We introduce and describe context specification as a process to support and inform the deployment decision making process. Context specification turns diffuse stakeholder perspectives about what matters in a given setting into clear, named constructs—explicit definitions of the properties, behaviors, and outcomes that evaluations aim to capture—so they can be observed and measured in context. The process serves as a foundational roadmap for evaluating what AI systems are likely to do in the deployment contexts that organizations actually manage. I Introduction and Motivation: From Benchmarks to Decision-Grade Evaluation As organizations adopt AI systems at increasing scale, decision makers are being asked to justify whether and how these systems should be deployed and used in their operational settings. The most consequential impacts of AI unfold in real contexts and over time, reshaping organizational processes and incentives and producing a growing list of downstream societal impacts [ 28 ] . Many of these impacts arise from how people adapt to, misuse, or over‑rely on AI in specific settings—factors that status quo evaluation methods largely overlook as they remain oriented toward model capabilities and optimization [ 4 , 30 , 25 , 32 ] . Meanwhile, stakeholders outside the stack aim to answer decision-critical questions about AI’s real‑world value and risk, such as: “What do we need to know about how this system behaves in our setting so that we can decide whether to adopt it, where it should be used, and what guardrails it requires?” When evaluation outcomes correlate poorly with what actually materializes in deployment environments, these stakeholders are left with little visibility into whether an AI deployment will generate genuine value or merely shift burdens to new parts of the workflow. In this paper, the term ”deployment” refers to the adoption and implementation of AI tools in users’ settings. I-A The Need for Well-Defined Constructs Addressing the gap between current evaluation outcomes and what stakeholders need requires a shift from generic performance metrics toward explicitly defining the outcomes and behaviors that matter in a given deployment environment, and translating them into concrete measurement targets–also referred to as ”constructs”. This process of turning diffuse ideas into precise, measurable constructs is known as construct systematization and is widely used in other measurement domains [ 6 ] . Existing approaches to construct development for AI evaluation fall into several categories. Some benchmarks overlook constructs entirely, treating scores as self-interpreting without specifying what real-world concept they are meant to capture [ 4 , 25 ] . Others import constructs from model tuning and optimization processes inside the AI stack, where labels like “reasoning” or “helpfulness” are treated as self-evident concepts rather than being fully defined for repeatable observation and measurement [ 6 ] . A complementary set of methods such as co-design, participatory techniques, and human-centered design are used to surface rich, locally grounded constructs around values, harms, and preferences, but primarily to support the design of systems rather than their evaluation [ 36 , 8 , 18 , 12 ] . A central challenge for these methods is whether the constructs we use to design and develop AI bear a stable relationship to what actually materializes in real-world settings, or whether internal metrics are drifting proxies that fail to account for downstream impacts on people and institutions. Constructs that connect well to real-world settings can be leveraged for evaluation, but even formalized constructs such as fairness remain under-examined with respect to whether model-level disparity measures reliably track who benefits or is burdened in the real-world [ 14 , 26 , 20 , 30 , 3 ] . When measures do not faithfully track materialized outcomes, deployment and adoption decisions are effectively being made on numbers that look rigorous but offer weak guidance about when a system is safe to rely on, where it is likely to fail, or which communities will bear the risks. Well specified constructs give organizations a stable reference point for assessment, enabling a shift away from trend-driven experimentation toward a clearly articulated destination and set of landmarks for whether, when, and how to deploy AI technologies. I-B Thesis and Problem Statement: Why Context Specification is Foundational Context specification can be understood as a subtype of construct systematization that remains explicitly focused on “what matters” to those in the deployment context. Rather than being driven by model developers’ internal objectives, context specification is carried out for, by, and in coordination with stakeholders outside the AI stack [ 27 ] . In doing so, it directs construct work toward real-world evaluation targets—how systems actually perform in use—rather than toward model-centric abstractions [ 1 , 6 ] . To support valid deployment‑relevant claims—especially about downstream outcomes—constructs elicited through this process must be described in ways that are technically precise, neutral, collectively informed, and unambiguous so that different actors cannot reasonably interpret them in incompatible ways. Many organizations adopting AI lack dedicated evaluation capabilities and are therefore left to rely on whatever tools and metrics are readily available. Those tools are rarely designed for the questions and settings relevant to organizations deciding whether to deploy, adopt or use AI for their own use cases, so evaluation outcomes do not translate cleanly into answers about whether a system is safe, usable, or valuable in context [ 37 ] . At the same time, the evaluation community more broadly lacks a stable, rigorous way to define the real-world concepts of interest to AI adopters and users. Benchmarks provide performance snapshots under controlled conditions; participatory and design‑oriented methods surface values and preferences; model‑internal metrics track optimization objectives. Yet none of these, on their own, ensure that what is measured corresponds to what ultimately materializes in deployment. We argue that systematic context specification is the foundational step for evaluating AI in the real-world. It translates stakeholder priorities—what matters to them in practice—and use contexts into evaluable constructs—explicit, named descriptions of properties, behaviors, and outcomes that can be observed and measured in a given setting. By making these constructs explicit and linking them to candidate observables, context specification creates the conditions for systematic knowledge rather than isolated performance claims, and delivers stakeholders the information they need to inform go/no-go decisions. This allows stakeholders outside the stack to gain direct knowledge about what matters in their own use cases so they can prioritize, manage, and govern AI deployments in ways that align with their goals and constraints, and to focus their resources on problems that actually materialize in practice rather than on abstract or hypothesized risks. Without systematic context specification, evaluation tends to collapse into three recurring problems. First, effects are misattributed: observed outcomes are interpreted as model performance when they may arise from

Read the complete text