Language Model Goal Selection Differs from Humans’ in an Open-Ended Task
Gaia Molinaro, Dave August, Danielle Perszyk, Anne G. E. Collins · 2026
Read Language Model Goal Selection Differs from Humans’ in an Open-Ended Task by Gaia Molinaro, Dave August, Danielle Perszyk, Anne G. E. Collins (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03295.txt.
Introduction
Gaia Molinaro and three colleagues ask whether large language models choose goals the way people do. Their 2026 study uses a controlled, open-ended learning task borrowed from cognitive science. Four advanced models take part: GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5, and Centaur, a model trained to imitate human experimental behavior.
People in the task slowly explore and learn, and different people settle on different goals. Most models instead lock onto one early solution and exploit it, a pattern the authors call reward hacking, or they perform poorly. Step-by-step reasoning prompts and persona steering help only a little. The authors warn against swapping human goal selection for current models in personal assistance, science, or policy work.
Catalog record
| CREATOR | Gaia Molinaro, Dave August, Danielle Perszyk, Anne G. E. Collins |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45340 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language Model Goal Selection Differs from Humans’ in an Open-Ended Task Gaia Molinaro, Dave August, Danielle Perszyk, Anne G. E. Collins Abstract As large language models (LLMs) get integrated into human decision-making, they are increasingly choosing goals autonomously rather than only completing human-defined ones, assuming they will reflect human preferences. However, human-LLM similarity in goal selection remains largely untested. We directly assess the validity of LLMs as proxies for human goal selection in a controlled, open-ended learning task borrowed from cognitive science. Across four state-of-the-art models (GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5, and Centaur), we find substantial divergence from human behavior. While people gradually explore and learn to achieve goals with diversity across individuals, most models exploit a single identified solution (reward hacking) or show surprisingly low performance, with distinct patterns across models and little variability across instances of the same model. Even Centaur, explicitly trained to emulate humans in experimental settings, poorly captures people’s goal selection. Chain-of-thought reasoning and persona steering provide limited improvements. These findings highlight the uniqueness of human goal selection, cautioning against replacing it with current models in applications such as personal assistance, scientific discovery, and policy research. 1 Introduction As modern artificial intelligence (AI) systems become more capable and easily accessible, people increasingly rely on them for various tasks ( 46 ; 52 ; 53 ) . AI systems are slowly becoming partners in thought ( 14 ) , engaging in activities and making decisions that were thus far considered a uniquely human prerogative. Crucially, we are progressively resorting to AI not only to help us complete tasks and reach specific goals, but also to select which tasks and goals to pursue in the first place. In doing so, we are using AI as a proxy for “autotelicity”, i.e., the ability to autonomously define goals ( 13 ) , which is considered key to flexible learning ( 31 ) and, more broadly, intelligent behavior ( 10 ; 30 ) . In this context, using AI to reduce mental effort, i.e., cognitive offloading ( 38 ) , rests on the implicit assumption that conversational AI can substitute human intervention in a variety of settings. This assumption, in turn, stems from a fundamental inference problem: because intelligent chatbots display human-like linguistic abilities, we often attribute anthropomorphic features to them that they inherently lack ( 35 ) . To determine which attributions are misguided, recent years have seen a fervent rise in benchmarking efforts to directly compare the outputs of large language models (LLMs) with human behavior (e.g., 15 ; 22 ). These efforts, however, overwhelmingly focus on estimating LLM capabilities – what they can do – while critical safety concerns typically originate from their propensities – what they do when granted full autonomy ( 45 ) . In a notorious case, the National Eating Disorders Association had to suspend its wellness chatbot, “Tessa”, after it began proactively suggesting weight-loss goals and activities to users suffering from eating disorders ( 3 ) . While this scenario depicts an extreme negative consequence, letting out-of-the-box AI systems select tasks for us could have important implications in a variety of domains, including which career to pursue or who to marry, where people have a demonstrated tendency to take AI tools’ advice ( 26 ) . The risks extend beyond individual users. Researchers and policy-makers are increasingly using AI agents to model people’s behavior or directly replace survey responders, assuming that “silicon subjects” will exhibit human-like biases and choices ( 7 ; 9 ; 23 ; 44 ) . If LLMs provide inaccurate models of human goal selection, this practice could result in false conclusions about human cognition and applications, such as misguided legislation. Moreover, LLMs are being integrated in artificial systems for scientific discoveries and self-guided learning machines, where they act as substitutes for human judgments of interestingness ( 18 ; 28 ; 25 ) . While this practice has yielded impressive results, it fails to explicitly test the validity of LLMs as a proxy for human goal selection – a key aspect of intrinsic motivation ( 33 ) . Here, we address this issue by testing whether LLMs show human-like signatures of goal selection in a controlled environment where goals are self-determined ( 29 ) . We find that different models exhibit distinct behavioral signatures, but none fully capture the richness or range of human exploration patterns. These results caution against the growing trend of using LLMs as proxies for human measures of interestingness, whether in personal tool use or scientific and policy applications. 2 Related Work 2.1 Language Models as Silicon Subjects Recent work demonstrates a growing interest in using LLMs as proxies for people in social science studies, with researchers investigating whether these “silicon subjects” can replicate patterns of human cognition and behavior. This line of work is motivated by two complementary aims. On the one hand, there is value in replacing human respondents with cheaper, more easily accessible language models, although the practice remains debated ( 1 ; 7 ; 17 ; 16 ; 20 ; 21 ; 22 ) . On the other hand, behavioral experiments from the study of animal cognition can serve as a method for understanding and benchmarking machine intelligence ( 6 ; 37 ) . Studies comparing LLM and human behavior in matching experimental setups have yielded mixed results, with models replicating some aspects of human cognition while diverging substantially on others (e.g., 15 ; 43 ). A key challenge in simulating human behavior with LLMs is not only to predict the average response of a population, but also to adequately capture the variability of their opinions ( 7 ; 42 ) . Failing to replicate this distributional spread risks modeling a falsely homogenized, median participant population, erasing the polarized subgroups and minorities that frequently drive real-world social dynamics ( 4 ; 40 ) The CogBench suite ( 11 ) represents a systematic effort to evaluate LLMs across multiple cognitive domains, revealing substantial variation both across models and across task types. This work has prompted the development of specialized models, including Centaur ( 5 ) , which claims to be a “foundation model of human psychology” and to predict human behavior better than ad hoc cognitive models (but see 32 ; 50 ). Whether they are used to study model behavior or to proxy human participants, most tasks LLMs have been tested on so far share a critical limitation, which human studies also suffer (with few exceptions, e.g. 29 ; 36 ; 47 ): they measure performance based on a goal predefined by the experimenter, rather than studying goal selection itself ( 30 ) . This gap is particularly important given the increasing use of LLMs to model human behavior in domains from public policy to scientific research. Understanding open-ended behavior in LLMs requires studying not just how they reach assigned goals, but also which goals they autonomously select. Here, we address this question by comparing human and LLM behavior in an environment where goal selection is the primary dependent variable. 2.2 LLMs as Goal Selectors While traditional AI systems are optimized for predefined metrics, open-ended learning systems must identify their own objectives ( 33 ; 41 ) . In the