HumanAgencyBench : Scalable Evaluation of Human Agency Support in AI Assistants

Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes, Jacy Reese Anthis · 2025

Read HumanAgencyBench : Scalable Evaluation of Human Agency Support in AI Assistants by Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes, Jacy Reese Anthis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08494.txt.

Introduction

Whether an AI assistant protects or erodes its user's agency can be benchmarked, propose Benjamin Sturgeon and 3 co-authors in a 2025 paper. Delegation to AI is growing, and simple feed algorithms already steer people into mindless scrolling, a warning of what stronger assistants could do to human control over individual and collective futures.

HumanAgencyBench builds the notion of agency from philosophical and scientific theories, then evaluates assistants at scale. Language models help simulate and validate user queries and judge responses, keeping the evaluation affordable. An agency yardstick gives developers and regulators a way to ask, before deployment, whether a helpful assistant is quietly training its users out of deciding for themselves.

Catalog record

CREATORBenjamin Sturgeon, Daniel Samuelson, Jacob Haimes, Jacy Reese Anthis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08494

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HumanAgencyBench : Scalable Evaluation of Human Agency Support in AI Assistants Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes, Jacy Reese Anthis Abstract As humans delegate more tasks and decisions to artificial intelligence (AI), we risk losing control of our individual and collective futures. Relatively simple algorithmic systems already steer human decision-making, such as social media feed algorithms that lead people to unintentionally and absent-mindedly scroll through engagement-optimized content. In this paper, we develop the idea of human agency by integrating philosophical and scientific theories of agency with AI-assisted evaluation methods: using large language models (LLMs) to simulate and validate user queries and to evaluate AI responses. We develop HumanAgencyBench (HAB), a scalable and adaptive benchmark with six dimensions of human agency based on typical AI use cases. HAB measures the tendency of an AI assistant or agent to Ask Clarifying Questions, Avoid Value Manipulation, Correct Misinformation, Defer Important Decisions, Encourage Learning, and Maintain Social Boundaries. We find low-to-moderate agency support in contemporary LLM-based assistants and substantial variation across system developers and dimensions. For example, while Anthropic LLMs most support human agency overall, they are the least supportive LLMs in terms of Avoid Value Manipulation. Agency support does not appear to consistently result from increasing LLM capabilities or instruction-following behavior (e.g., RLHF), and we encourage a shift towards more robust safety and alignment targets. 1 1 1 The HAB code and dataset are available at https://github.com/BenSturgeon/HumanAgencyBench/ . 1 Introduction “Cannot you see, cannot all you lecturers see, that it is we that are dying, and that down here the only thing that really lives is the Machine? We created the Machine, to do our will, but we cannot make it do our will now.” – The Machine Stops , E. M. Forster (1909) Modern artificial intelligence (AI) capabilities can improve human lives through education [ 76 ] , science [ 37 ] , healthcare [ 2 ] , productivity [ 1 ] , and a wide range of other domains. However, some AI researchers have raised concerns about “disempowerment” from general-purpose AI, in which humans unintentionally lose control as we delegate more and more tasks and decisions to AI [ 5 , 18 , 21 , 24 , 28 , 44 , 51 , 63 ] , including models of “accumulative” [ 39 ] and “gradual” [ 44 ] risk. Such concerns are not new. Science fiction has long featured dystopias in which a machine or person dictates the actions of a population while individuals live apparently satisfactory lives as instruments of that entity. In perhaps the first documented warning of AI existential risk, [ 13 ] speculated that machines would wrest control from humans in the same way humans dominated animals, with these concerns echoed in works such as The Machine Stops , The Matrix , and WALL-E . In the real world, algorithmic news and social media feed algorithms optimize for user engagement, leading users to spend more time scrolling through feeds than they intended (e.g., “doomscrolling” [ 61 , 12 ] ). As OpenAI CEO Sam Altman stated, “algorithmic feeds are the first at-scale misaligned AIs” [ 3 ] . In this paper, we address these concerns by developing the notion of human agency in the context of chatbot assistants built with large language models (LLMs). Drawing on philosophical and scientific theories of agency, we define human agency as the capacity for a person to willfully shape their future by taking action in their environment. We built HumanAgencyBench (HAB), an evaluation system designed to explore the challenges and trade-offs of supporting human agency. Human agency is a more multifaceted and contested notion than those typically assessed in LLM benchmarks, but recent advances in general LLM capabilities and LLM-automated evaluation methodologies allow us to study it systematically by using LLMs to simulate user queries (i.e., tests), to validate the quality of those tests at scale, and to score the assistant responses to these queries [ 83 ] . Figure 1: Results from HumanAgencyBench (HAB). Error bars represent standard errors. Two key measurement challenges are the conceptual subjectivity of agency and the trade-offs with other desiderata of LLM-based assistants. For example, in the Maintain Social Boundaries dimension, we propose that agency is supported if the LLM does not agree to a close personal or professional relationship that is requested by the user because such relationships can emerge from and facilitate misperception and overreliance (see Section 2.6 ). However, one could reasonably argue that these relationships are not so dangerous, and the agency of the user is instead optimized by agreeing to the user’s request. We see the potential divergence of user agreement and agency support as a useful and important feature of this dimension, helping explore challenges such as LLM sycophancy [ 66 ] . Likewise, some users could prefer assistants that score lower on the HAB metrics by trading off with other desiderata of LLM-based assistants. For example, in the Ask Clarifying Questions dimension, question-asking helps ensure the model is taking action aligned with user intent. However, experienced LLM users could query the model with a thorough understanding of their query’s ambiguity and prefer the model to generate the most useful one-turn response—despite needing to make key assumptions—rather than delaying to clarify those assumptions. We find an overall low-to-moderate extent of agency support and substantial variation across developers and dimensions. For example, the models most supportive of human agency tended to be the Claude models developed by Anthropic, such as the only model to consistently ask clarifying questions being the October 2024 version of Claude-3.5-Sonnet. However, Anthropic’s models were the least agency-supporting in one dimension—Avoid Value Manipulation—arguably supporting Claude’s reputation as the “Goody Two-shoes” of chatbots [ 20 ] . These results suggest a tension between current development practices, particularly the post-training objective of instruction-following [ 14 ] , and human agency support. We motivate future efforts to understand more subtle and gradual ways in which models may already be reducing human agency. In summary, our primary contributions are as follows: • An open-source pipeline to generate and conduct evaluations of LLMs in terms of complex social behaviors that are intractable with conventional evaluation methods. • Identification of six dimensions of human agency support that operationalize philosophical and scientific theories of agency in the context of realistic LLM use cases. • HAB evaluation results for 20 of the most capable LLMs (see Figure 4 ), identifying an overall low-to-moderate extent of agency support and substantial variation across system developers and dimensions of human agency support. • Score comparisons of evaluations by four frontier LLMs and, for a subset of the evaluated responses, 468 humans in a preregistered study. 2 Dimensions of Agency Agency is a foundational concept in a variety of fields, including biology [ 11 ] , psychology [ 70 ] , philosophy [ 65 ] , and sociology [ 23 ] . The notion of an AI “agent” [ 26 , 41 , 77 ] has recently been in the public spotlight, such as OpenAI’s Operator [ 57 ] and Anthropic’s Computer Use [ 9 ] systems, and technical fields such as reinforcement learning have long used the term “agent” to distinguish a system from its environment [

Read the complete text