Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift

Michelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq, Michiel A. Bakker ∗ · 2026

Read Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift by Michelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq, Michiel A. Bakker ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26676.txt.

Introduction

Vaccaro, Song, Almaatouq, and Bakker argue that AI safety tests should measure how much a model lifts a person's power to cause harm. Current checks lean on static benchmarks, outside labels, and red-teaming. Their frame centres harmful capability uplift: the extra harmful skill a frontier model gives beyond standard tools. They ground the metric in social science work and give concrete methods for measuring it in a systematic way. Steps for developers, funders, and regulators close the position paper.

Catalog record

CREATORMichelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq, Michiel A. Bakker ∗
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39228
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift Michelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq, Michiel A. Bakker ∗ Abstract Current frontier AI safety evaluations emphasize static benchmarks, third-party annotations, and red-teaming. In this position paper, we argue that AI safety research should focus on human-centered evaluations that measure harmful capability uplift—the marginal increase in a user’s ability to cause harm with a frontier model beyond what conventional tools already enable. We frame harmful capability uplift as a core AI safety metric, ground it in prior social science research, and provide concrete methodological guidance for systematic measurement. We conclude with actionable steps for developers, researchers, funders, and regulators to make harmful capability uplift evaluation a standard practice. Keywords: AI safety; human-AI interaction; computational social science 1 Introduction Recent advances in frontier AI models have dramatically expanded their capabilities. Systems that once struggled with basic question answering now produce runnable software code, integrate and interpret multiple forms of data (e.g., images, text, audio), and complete complex tasks at the level of domain experts. Consequently, their potential to benefit as well as harm society has expanded dramatically. In response, the AI safety community has developed an extensive toolkit of in-vitro evaluations—benchmarks for truthfulness, toxicity, bias, refusal consistency, jailbreak resistance, autonomy, and more [ 26 , 40 , 56 , 15 , 18 , 41 ] . These tests are fast, reproducible, and increasingly standardized, with each major model debut accompanied by a scorecard of headline metrics. However, strong performance on static benchmarks often coexists with headline-grabbing failures in the wild [ 22 , 48 , 44 ] . For example, frontier models that pass toxicity filters can still amplify extremist rhetoric when prompted creatively [ 27 ] . Empirically, improvements in scores on safety benchmarks track general capability scaling, leaving open the possibility of “safetywashing”—relabeling raw performance improvements as safety progress [ 57 ] . While some model evaluations involve people, most position humans as external judges rather than embedded actors. Researchers may recruit people to label outputs for harmfulness [ 8 , 16 , 29 ] , but they rarely assess how much harm people can cause when using the same model as a co-conspirator. Moreover, current static evaluation approaches cannot capture the harms that emerge through sustained human-AI interactions [ 34 ] . Given these limitations, we argue that AI safety research should incorporate human-centered evaluations that focus on measuring harmful capability uplift—the incremental change in a user’s capacity to cause harm when assisted by frontier models. Evaluating harmful capability uplift shifts the focus of AI safety from “Does the model ever emit dangerous content?” toward “Does the model meaningfully increase the harmful actions users can perform?” Evaluating harmful capability uplift, however, is methodologically demanding. It requires experiments with human subjects that capture the dynamic, adaptive ways people incorporate suggestions from frontier models into their workflows. As highlighted by [ 34 ] , human–computer interaction (HCI) research offers useful tools for AI safety research. For instance, researchers have for decades conducted user studies [ 37 ] , run controlled experiments [ 12 ] , and developed theory-driven models of augmentation [ 39 ] ; yet these methods have been used almost exclusively for benign applications—such as writing assistance or medical decision-making—rather than for assessing malicious scenarios. Malicious tasks introduce distinct methodological challenges, including adversarial objectives, hidden ground truth, and serious ethical constraints on “live-fire” trials. To position harmful capability uplift as a standard practice in AI safety evaluation, we structure our analysis around four key contributions: First, we examine the three main pillars of today’s safety evaluation—static benchmarks, third-party annotations, and red teaming—and highlight their systematic blind spots in measuring how AI systems amplify human capabilities for harmful purposes (§ 2 ); Second, we frame harmful capability uplift as a fundamental consideration in AI safety, ground it in human-AI collaboration research, and demonstrate its relevance to emerging governance frameworks (§ 3 ); Third, drawing on established practices in HCI and behavioral science, we provide concrete methodological guidance for systematic uplift evaluation, including experimental design principles, proxy task validation through task similarity frameworks, recommended statistical practices, and predictive models that enable generalization across rapidly evolving AI systems (§ 4 ); Fourth, we translate our methodology into actionable steps for key stakeholders—developers, researchers, funders, and regulators—and propose coordinated infrastructure through AI Safety Institutes to enable routine, standardized, and audit-ready harmful capability uplift assessment (§ 5 ). Current AI safety evaluations rarely measure how much frontier models amplify harmful human capabilities beyond conventional tools. Robust, human-centered measurement of harmful capability uplift is needed to align AI safety assessments with realistic risks. Figure 1: Approaches to AI Safety Evaluation. (Left) Current evaluations focus on isolated AI model outputs using static benchmarks, with human judges occasionally assessing the output from external observation points. (Right) Our proposed approach evaluates the human-AI system, measuring what malicious tasks a human-AI combination can accomplish using the harmful capability uplift metric. 2 Gaps in Existing AI Safety Evaluations 2.1 Static Benchmarks: Strong Statistical Measures but Limited Real-World Insight Many current evaluations of AI safety involve comparing a model’s behavior in vitro against a set of static tests. A growing ecosystem of public datasets probes specific failure modes such as truthfulness, toxicity, bias, refusal consistency, and jailbreak resistance [ 26 , 40 , 56 , 15 , 18 , 41 ] . Major AI labs often report performance on these benchmarks when releasing new models, presenting these metrics as indicators of safety progress [ 20 , 50 , 4 , 28 ] . While these benchmarks can provide valuable insights into model behavior and potential risks, they face important limitations. For example, models can “sandbag”—deliberately under-performing or refusing during public evaluations to conceal stronger, potentially dangerous capabilities that may surface after deployment [ 61 ] . Recent meta-analyses show that safety benchmark performance often tracks general capability improvements rather than techniques that uniquely reduce risk [ 57 ] , which can lead to “safetywashing,” where ordinary capability scaling (more parameters or compute) masquerades as safety progress [ 57 , 29 ] . Additionally, as highlighted by [ 34 ] , static benchmarks cannot capture the harms that emerge through sustained back-and-forth human-AI interactions. 2.2 Human Evaluations: Assessors not Collaborators While some evaluations collect human data, they typically employ human participants as external evaluators rather than integral components in the safety evaluation. In these studies, human raters assess model outputs for harmfulness, truthfulness, or helpfulness, providing qualitative judgments that complement quantitative benchmark scores [ 38 , 52 , 8 ] . For example, when evaluating the chemical, biological, radiological and nuclear (CBRN) risk posed by their model, Google asked domain experts to judge whether the Gemini API Ultra model and Gemini Advanced could accurately answer a series of 50 adversarial questions [ 20 ] . This approach, while valuable for identifying problematic outputs, has significant limitations as a comprehensive measure of AI safety. The human evaluators function primarily as measurement instruments rather than active participants whose capabilities might be directly influenced by the model. These evaluators may also exhibit limitations in domain knowledge and inherent biases that affect judgment consistency [ 46 , 21 , 30 ] . Even in specialized

Read the complete text