Alignment has a Fantasia Problem

Nathanael Jo, Zoe De Simone, Mitchell Gordon, & Ashia Wilson · 2026

Read Alignment has a Fantasia Problem by Nathanael Jo, Zoe De Simone, Mitchell Gordon, & Ashia Wilson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21827.txt.

Introduction

Jo and three colleagues name a failure mode of AI assistance: Fantasia interactions. When a user's goals are still abstract, instruction-tuned systems often jump straight to a final output, short-circuiting the thinking through which goals would sharpen. The user then spends more time revising, or settles for a worse outcome, losing agency in the task. The paper argues alignment research should optimise how cognitive responsibility is shared in an interaction, and sketches a research agenda for training and evaluating models toward that goal.

Catalog record

CREATORNathanael Jo, Zoe De Simone, Mitchell Gordon, & Ashia Wilson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61965
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21827

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Alignment has a Fantasia Problem Nathanael Jo, Zoe De Simone, Mitchell Gordon, & Ashia Wilson Abstract Modern AI assistants are trained to follow instructions, implicitly assuming that users can clearly articulate their goals and the kind of assistance they need. Decades of behavioral research, however, show that people often engage with AI systems before their goals are fully formed. When AI systems treat prompts as complete expressions of intent, they can appear to be useful or convenient, but not necessarily aligned with the users’ needs. We call these failures Fantasia interactions . We argue that Fantasia interactions demand a rethinking of alignment research: rather than treating users as rational oracles, AI should provide cognitive support by actively helping users form and refine their intent through time. This requires an interdisciplinary approach that bridges machine learning, interface design, and behavioral science. We synthesize insights from these fields to characterize the mechanisms and failures of Fantasia interactions. We then show why existing interventions are insufficient, and propose a research agenda for designing and evaluating AI systems that better help humans navigate uncertainty in their tasks. Figure 1: Diagram describing a Fantasia interaction, including behavioral sources and failure modes. 1 Fantasia Interactions In a scene from the 1940 film Fantasia , Mickey Mouse plays a sorcerer’s apprentice who needs to clean his master’s room. With access to a book of spells, he enchants a broom, instructing it to carry water and clean the room. Mickey falls asleep as the broom does its job, but the broom continues hauling bucket after bucket of water until the room floods. Lacking any context beyond the instructions it was given, the broom faithfully executed the task to disastrous effect, and Mickey is left dealing with the consequences. This cautionary tale bears striking resemblance to many human-AI interactions today. Users often prompt AI systems quickly without fully articulating their goals, constraints, or broader context ( 68 ) . However, modern AI is optimized to respond helpfully to the prompt as written . While models sometimes reason about intent or ask clarifying questions, these behaviors are only weakly encouraged by current training methods. Like the enchanted broom, the model faithfully executes the instruction it is given, even when doing so produces behavior that is misaligned with the user’s ongoing intellectual or creative process. We define this phenomenon as a Fantasia interaction : a failure of human-AI coordination where the system commits to a concrete interpretation from a prompt that may only be an early, evolving signal of the user’s intent. Fantasia interactions may initially appear as prompting failures, but we argue that they reflect a more fundamental alignment problem. The core issue is not how clearly users express requests, but how AI systems interpret and optimize under incomplete and uncertain signals of human intent. In many real-world interactions, users’ goals may be constantly evolving, under-specified, or only partially articulated. As a result, when users approach a general-purose AI assistant, their prompts provide only an incomplete proxy for what the user ultimately cares about. For example, in Education (Figure 2 ), a student may ask an AI assistant to solve a problem set rather than help them understand missing concepts; in Writing assistance , an applicant may ask for a draft of a personal statement instead of support in shaping a compelling narrative; in Advice seeking , a user may ask for productivity advice thinking that that would solve their problem, even though there are deeper root causes to their lack of productivity. In each case, the prompt is reasonable in isolation, but because the AI assistant takes the request at face value and is quick to provide a solution, the interaction results in an outcome that ultimately does not address the user’s underlying needs. Our position is that alignment research should address Fantasia interactions by desgining AI systems that provide cognitive support to users. In practice, this means AI systems should help users articulate and refine their intent or goals over time. As we later argue, research towards this goal remains fragmented; machine learning (ML) works tend to treat humans as oracles or rational users—overlooking our many behavioral biases—while human-computer interaction (HCI) research does account for these biases but typically lacks scalable interventions for general-purpose systems. This gap motivates an interdisciplinary approach that integrates ML, interface design, and behavioral science, which underlies our perspective in this paper. Figure 2: Illustrative examples of Fantasia interactions caused by base LLMs (a) and existing alignment methods (b), as well as the desired interaction (c), across three domains: Education , Writing assistance , and Advice seeking . Structure of paper. We develop our argument in three parts. (i) We synthesize evidence from behavioral science, ML, and HCI to characterize Fantasia interactions and their downstream failures (Sections 2 - 4 ). (ii) We then show why existing ML and HCI interventions only partially address these failures in general-purpose models (Section 5 and 7 ). (iii) Finally, we propose a research agenda for better aligning and evaluating AI models (Sections 6 and 8 ). While significant challenges remain, our work motivates a rethinking of alignment as support for human cognition under uncertainty. 2 Human-side of Fantasia Interactions A Fantasia interaction is a failure on both the human side and the AI side. In this section, we cover the evidence and behavioral sources behind why humans often come to AI with under- or mis-specified instructions. We emphasize, however, that human and AI behavior mutually shape each other : AI design influences how users formulate requests just as much as user prompts constrain how AI systems respond. As a result, the boundary between the “human-side” and “AI-side” may be blurred, and we draw these connections explicitly where relevant. 2.1 Evidence A growing body of work in HCI suggests that humans frequently issue vague or underspecified prompts, leading to inefficient iteration ( 24 ) . These issues become more pronounced as tasks grow larger or more complex; when problems involve many interdependent decisions, users struggle to state goals, preferences, or success criteria upfront ( 15 ; 37 ) . Rather than reflecting poor planning, this behavior often arises because users are still forming their goals. People tend to prompt quickly and revise after observing failures, treating interaction as an exploratory process rather than a deliberative one ( 68 ) . 2.2 (Behavioral) Sources Why does this happen? Writing a well-specified prompt is not just a language or engineering problem, it is a metacognitive one. To specify what kind of help would be most useful, users must reflect on their goals, preferences, and uncertainties, which is cognitively expensive ( 16 ; 28 ) . Our argument is that AI systems create an unusually low-friction environment for action, encouraging users to skip this reflection and employ a brute force strategy. We outline three behavioral phenomena that contribute to this problem, though none of them are mutually exclusive: 1. Human decision-making is shaped by present bias and bounded rationality. People systematically prefer quick actions with immediate feedback over slower, more deliberate planning ( 29 ; 45 ) . They also rely on simplified heuristics when tasks feel complex or overwhelming ( 59 ) . For example, in the Education scenario, the

Read the complete text