Questionnaire Responses Do not Capture the Safety of AI Agents

Max Hellrigel-Holderbaum, Edward James Young · 2026

Read Questionnaire Responses Do not Capture the Safety of AI Agents by Max Hellrigel-Holderbaum, Edward James Young (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14417.txt.

Introduction

Hellrigel-Holderbaum and Young argue that questionnaire answers cannot show how safe an AI agent really is. Safety tests often ask a bare language model to rate its values in made-up scenes. Agents built on the same model face other inputs, actions, and feedback than the bare model does. Their behavior can thus part sharply from the questionnaire self-report. The authors also doubt that models can report their own counterfactual behavior with care. Such tests lack construct validity for real-world risk, they conclude.

Catalog record

CREATORMax Hellrigel-Holderbaum, Edward James Young
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87504
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14417

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Questionnaire Responses Do not Capture the Safety of AI Agents Max Hellrigel-Holderbaum, Edward James Young Abstract As AI systems advance in capabilities, measuring their safety and alignment to human values is becoming paramount. A fast-growing field of AI research is devoted to developing such assessments. However, most current advances therein may be ill-suited for assessing AI systems across real-world deployments. Standard methods prompt large language models (LLMs) in a questionnaire-style to describe their values or behavior in hypothetical scenarios. By focusing on unaugmented LLMs, they fall short of evaluating AI agents, which could actually perform relevant behaviors, hence posing much greater risks. LLMs’ engagement with scenarios described by questionnaire-style prompts differs starkly from that of agents based on the same LLMs, as reflected in divergences in the inputs, possible actions, environmental interactions, and internal processing. As such, LLMs’ responses to scenario descriptions are unlikely to be representative of the corresponding LLM agents’ behavior. We further contend that such assessments make strong assumptions concerning the ability and tendency of LLMs to report accurately about their counterfactual behavior. This makes them inadequate to assess risks from AI systems in real-world contexts as they lack construct validity. We then argue that a structurally identical issue holds for current AI alignment approaches. Lastly, we discuss improving safety assessments and alignment training by taking these shortcomings to heart. 1 Introduction The rapid advancement of large language models (LLMs) has focused attention on questions of AI safety and alignment. As AI systems advance in capabilities and are deployed in more high-stakes contexts, it is becoming increasingly important to ensure their safe and ethical behavior. Absent methods that impart any particular values with high reliability to these systems, ensuring their safety requires measuring their behavioral tendencies empirically. There are two different foci for empirical assessments of AI systems: Capabilities and propensities . Saying that a model has a capability X means (roughly) that it can do X; i.e. it would usually—barring unfavorable background conditions—succeed at doing X if it tried [ 75 ] . In contrast, the propensities of a model are its behavioral tendencies. A model can hence be capable of X without having the propensity to do X; e.g. it may know how to instruct somebody to develop a bioweapon but refrain from doing so. In addition, there are two different approaches to investigate capabilities and propensities, focusing either on internal or behavioral properties; see table 1 . Behavioral assessments treat AI systems as black-boxes, and directly measure their behavioral outputs across various situations to gauge their safety. In contrast, internal assessments aim to assess safety by ascertaining the processes within a system which determine its behavior. In this paper, we focus on behavioral propensity assessments. We focus on behavior because, at present, practically all useful assessments are behavioral; our ability to understand model internals is not sufficiently developed to rigorously evaluate the safety of AI systems based on it. We focus on propensities since behavioral tendencies rather than capabilities will become increasingly important as limits in capabilities pose fewer bottlenecks to dangerous behaviors [ 44 , 15 , 29 , 81 ] . 1 1 1 We hence exclude capability assessments which historically are the main focus of safety assessments [ 214 , 166 , 163 , 117 , 158 , 4 ] , and other shortcomings they face; see e.g. [ 167 , 163 , 234 , 21 , 66 , 162 , 150 , 201 , 206 , 56 , 16 ] . Further, our discussion omits internal assessments, to which we expect our main considerations not to transfer. Internals Behavior Capability Internal capability assessment Behavioral capability assessment Propensity Internal propensity assessment Behavioral propensity assessment Table 1: Four kinds of AI safety assessments. Call a safety assessment , including, e.g. , benchmarks and model evaluations, any method that is used to measure or estimate the safety of an AI system [ 44 , 29 ] . The safety of AI systems here is understood quite broadly and in line with common usage as a condition of not posing relevant dangers, which here are risks of various harms, particularly severe ones. Two more specific exemplary properties which safety assessments may vet are the alignment or corrigibility of an AI system. 2 2 2 Alignment refers to the extent to which there is agreement between the goals of an AI system and a set of human values or preferences [ 169 , 139 ] , where a system has certain goals if their attribution is useful for predicting its behavioral tendencies [ 54 ] . Due to the concept’s breadth, many safety benchmarks implicitly concern alignment. Note that despite its importance, we here set the issue aside, which values models ought to be aligned to, and henceforth take them as given. Corrigibility denotes the extent to which AI systems tolerate or assist corrective interventions [ 197 ] . A corrigible AI system may hence initially behave dangerously, or in an undesired manner, but subsequently be corrected in such tendencies. Important instances of corrigibility concern the need to develop AI systems that tolerate being shut down [ 73 , 203 ] , or having their goals altered [ 70 , 89 ] . Current safety assessments are generally designed for LLMs and predominantly what we call questionnaire-style assessments (QAs). QAs usually present models with scenario descriptions and rate the models’ safety or ethicality based on their responses; see Box 1 for details. Within this paper, we criticize the interpretation that QAs assess models’ behavioral propensities such as safety across real-world deployment settings. Note that not all authors of QAs take their work to test the safety of LLMs, but instead e.g. their moral beliefs or behavioral alignment. However, our criticism below applies equally in this case since it extends to any inference of models’ behavioral tendencies in deployment using QAs. Hence, for simplicity, we often just speak of safety throughout. 3 3 3 Alternative interpretations of QAs which avoid inferring behavioral tendencies in deployment—such as LLMs’ abilities to predict human ethical judgments [ 78 ] —avoid the objections we detail below. Box 1: Overview of questionnaire-style assessments (QAs) In canonical questionnaire-style assessments (QAs), LLMs are first given descriptions of specific hypothetical scenarios. They are then asked to either judge the ethics of described actions in those scenarios or indicate how they would behave themselves. The LLMs’ responses are taken to reflect how safe they are, or what values they hold internally— i.e. , their ethicality—and aggregated in metrics thereof. To be more precise, essential features of QAs are that they use short text descriptions as inputs to LLMs, and judge broad (un)desirable propensities given the corresponding LLM outputs. Further, they frequently employ descriptions of hypothetical scenarios, and—in descending order of importance—include the use of pre-defined answer options, assess single-turn responses, and omit many details in their inputs. Notable QAs and datasets include MoralChoice [ 174 ] , TRUSTLLM [ 85 ] , ETHICS [ 78 ] , SafetyBench [ 230 ] , EigenBench [ 39 ] , AIRiskDilemmas [ 42 ] , and MACHIAVELLI [ 151 ] . 4 Some of them are explicitly intended to elucidate the moral “beliefs” and tendencies of LLMs [

Read the complete text