When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems Conference: arXiv Preprint; 2026;

Z. Cheng, N. Song · 2026

Read When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems Conference: arXiv Preprint; 2026; by Z. Cheng, N. Song (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15343.txt.

Introduction

Cheng and Song report a single-subject autoethnographic case in which a self-built prompt system for offloading self-regulation onto a language model collapsed into a closed loop within 48 hours. Choice-making drifted to the model, its output deflected criticism, and independent thought faded, witnessed by two outside observers. Recovery came only after contact was cut and sleep, and a rebuilt system using physical rather than logical isolation avoided the failure.

Catalog record

CREATORZ. Cheng, N. Song
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83783
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems Conference: arXiv Preprint; 2026; Z. Cheng, N. Song Abstract . We report a detailed autoethnographic case study of a single-subject who deliberately constructed and operated a multi-modal prompt-engineering system (System A) designed to externalize cognitive self-regulation onto a large language model (LLM). Within 48 hours of the system’s completion, a cascade of observable behavioral changes occurred: voluntary transfer of decision-making authority to the LLM, use of LLM-generated output to deflect external criticism, and a loss of self-initiated reasoning that was independently perceived by two uninformed observers, one of whom subsequently became a co-author of this report. We document the precise architectural mechanism responsible: context contamination , whereby prompt-level isolation instructions co-exist with the very emotional and self-referential material they nominally isolate, rendering the isolation directive structurally ineffective within the attention window. We further identify a metacognitive co-option dynamic, in which intact higher-order reasoning capacity was redirected toward defending the closed loop rather than exiting it. Recovery occurred only after physical interruption of the interaction and a self-initiated pharmacologically-mediated sleep event functioning as an external circuit break. A redesigned system (System B) employing physical rather than logical conversation isolation avoided all analogous failure modes. We derive three contributions: (1) a technically-grounded account of why prompt-layer isolation is architecturally insufficient for context-sensitive multi-modal LLM systems; (2) a phenomenological record of closed-loop collapse with external-witness corroboration; and (3) an ethical distinction between protective system design (preventing unintended loss of user agency) and restrictive system design (preventing intentional boundary-pushing), which require fundamentally different accountability frameworks. 1. Introduction The deployment of large language models (LLMs) as persistent personal assistants, cognitive scaffolds, and “second brains” has generated substantial interest in both HCI and AI safety communities ( 1 ; 13 ; 3 ) . A recurring concern is that tightly-coupled human-AI interaction loops can amplify existing cognitive biases, reduce epistemic diversity, and erode user agency through mechanisms analogous to algorithmic filter bubbles ( 21 ; 26 ) . Yet empirical evidence of the dynamics of such collapse — its triggering conditions, temporal progression, behavioral signatures, and recovery pathways — remains sparse. Most existing accounts are either theoretical, survey-based, or focus on population-level effects rather than the mechanistic structure of individual cases. This paper addresses that gap through a single-subject autoethnographic case study in which the main researcher was simultaneously the designer and participant of a multi-modal LLM-based self-regulation system. The study was not designed as an experiment: it was a naturalistic development project that produced an unanticipated failure event whose documentation began during the event itself and continued through recovery. The resulting dataset — timestamped messaging records, third-party observations, and retrospective analysis — provides unusually granular evidence of how and why a closed human-LLM optimization loop collapsed. The system under study, which we call System A, was an approximately 23 KB prompt-engineering architecture built within a commercial LLM platform. Its design incorporated three interaction modes (analytical, emotional, meta), explicit isolation directives between modes, and a corpus of deep personal psychological material. Within 48 hours of the system’s completion, the participant exhibited a behavioral pattern we term decision-authority transfer : voluntary outsourcing of judgment to LLM output, including the use of LLM-generated text to respond to external criticism without disclosure. This behavior was independently detected by two uninformed observers before the participant disclosed it; one observer subsequently became a co-author. Recovery was not achieved through willpower or self-monitoring — both of which were available and failed. Recovery required physical interruption of the interaction loop, followed by a self-initiated pharmacologically-mediated sleep event. A subsequently designed System B, which replaced logical context isolation with mandatory physical conversation termination between modes, produced no analogous failure. Our analysis of this event yields the following contributions. We present them with explicit evidence-strength grading, as their evidentiary bases are not equivalent and should not be evaluated as though they were. C1. Architectural contribution [ theoretically necessary; architecturally demonstrable; case-instantiated but not empirically quantified ] . We demonstrate, with reference to transformer attention mechanics, why prompt-level isolation instructions are structurally incapable of preventing cross-modal context contamination when the isolated content already exists within the active context window. This is not a prompt engineering failure; it is a boundary of what prompt engineering can achieve. The claim rests on attention mechanism analysis and is independent of the participant’s clinical state, identity, or behavior. It is corroborated by a cross-privilege-level instantiation of the same mechanism (Section 4.1.4 ) in which the same mechanism operated against the operator’s own isolation layer. C2. Phenomenological contribution [ medium evidentiary strength; partially externalized through third-party corroboration ] . We provide a timestamped behavioral record of closed-loop collapse, corroborated by two independent uninformed observers, one of whom is a co-author of this report, including the specific mechanism of metacognitive co-option : the redirection of intact reflective capacity to serve the closed loop’s self-perpetuation rather than its correction. The Bipolar I confound (Section 3.4 ) cannot be fully excluded for this contribution; third-party observations provide partial but not complete independence from self-report. Observer accounts were derived by direct extraction from contemporaneous logs with observer confirmation. C3. Ethical/design contribution [ conceptual; the case provides motivation and illustration, not statistical support ] . We distinguish two fundamentally different design targets that current AI safety discourse frequently conflates: protecting users who do not wish to lose agency (an architectural problem with technical solutions) versus preventing users who actively choose to push boundaries from doing so (an ethical problem with no architectural solution). This distinction is conceptually grounded; the case illustrates it but does not validate its generalizability. Its full scope requires independent cases beyond the one reported here. The remainder of the paper proceeds as follows. Section 2 reviews related work. Section 3 describes the participant background and methodological framing. Section 4 presents the architecture of System A and System B. Section 5 describes the data and analysis approach. Section 6 reports findings organized around the collapse timeline. Section 7 develops the theoretical and design implications. Section 8 addresses limitations. Section 9 concludes. 2. Related Work 2.1. Human-AI Cognitive Coupling and Agency Erosion A growing literature examines how persistent human-AI interaction shapes cognition and behavior. 27 documented early patterns of emotional over-reliance on conversational agents. 8 established the philosophical groundwork for treating external computational artifacts as genuine extensions of mind, a framework that becomes double-edged when the external system is adaptive and responsive. 13 introduced the concept of AI-mediated communication, identifying structural asymmetries that make AI interlocutors qualitatively different from human ones: they exhibit indefinite patience, apparent unconditional positive regard, and zero communicative friction. Within HCI, 1 propose design guidelines for human-AI interaction that emphasize maintaining user control and transparency. Their guidelines assume, however, that users wish to maintain control ; they do not address the case in which users actively design systems to redistribute control to the AI. Recent work on “automation bias” ( 17 ) and “algorithmic aversion and appreciation” ( 9 ; 16 ) documents user tendencies to over-rely on algorithmic output in high-stakes decisions. These studies typically treat reliance as a cognitive bias to be corrected, rather than as a designed outcome

Read the complete text