Contextual Agentic Memory is a Memo, Not True Memory
Binyan Xu, Xilin Dai, Kehuan Zhang, [4pt] The Chinese University of Hong Kong, Hong Kong, China · 2026
Read Contextual Agentic Memory is a Memo, Not True Memory by Binyan Xu, Xilin Dai, Kehuan Zhang, [4pt] The Chinese University of Hong Kong, Hong Kong, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27707.txt.
Introduction
Xu, Dai, and Zhang argue current agentic memory systems implement lookup, not memory. Scratchpads, vector stores, retrieval-augmented generation, and context handling retrieve by similarity to stored cases, while weight-based memory applies abstract rules to unseen inputs. Treating lookup as memory, they warn, caps generalization on novel tasks no matter how much context or retrieval quality grows, and leaves agents open to lasting memory poisoning. Drawing on Complementary Learning Systems, biology pairs fast hippocampal storage with slow neocortical consolidation, while current agents implement only the first half.
Catalog record
| CREATOR | Binyan Xu, Xilin Dai, Kehuan Zhang, [4pt] The Chinese University of Hong Kong, Hong Kong, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53993 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27707
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Contextual Agentic Memory is a Memo, Not True Memory Binyan Xu, Xilin Dai, Kehuan Zhang, [4pt] The Chinese University of Hong Kong, Hong Kong, China Abstract Current agentic memory systems (vector stores, retrieval-augmented generation, scratchpads, and context-window management) do not implement memory: they implement lookup . We argue that treating lookup as memory is a category error with provable consequences for agent capability, long-term learning, and security. Retrieval generalizes by similarity to stored cases; weight-based memory generalizes by applying abstract rules to inputs never seen before. Conflating the two produces agents that accumulate notes indefinitely without developing expertise, face a provable generalization ceiling on compositionally novel tasks that no increase in context size or retrieval quality can overcome, and are structurally vulnerable to persistent memory poisoning as injected content propagates across all future sessions. Drawing on Complementary Learning Systems theory from neuroscience, we show that biological intelligence solved this problem by pairing fast hippocampal exemplar storage with slow neocortical weight consolidation, and that current AI agents implement only the first half. We formalize these limitations, address four alternative views, and close with a co-existence proposal and a call to action for system builders, benchmark designers, and the memory community. 1 Introduction Modern LLM agents remember by writing. MemGPT pages information in and out of context like an OS ( 32 ) ; Generative Agents record every observation in a memory stream ( 34 ) ; Reflexion stores verbal self-critiques in an episodic buffer ( 37 ) ; Voyager accumulates skills as code in a vector database ( 40 ) . Every major agentic framework treats retrieval as the default persistence mechanism ( 15 ) , and the implicit promise is that richer external stores will eventually produce agents with richer inner lives. The pivot to retrieval was not irrational: external stores are reversible, auditable, and safe to deploy—genuinely valuable properties under production constraints. Our argument is not that retrieval was the wrong choice to ship; it was the right choice to ship. But the field has since conflated “best engineering compromise available” with “sufficient substitute for learning,” and this conflation is now a category error with provable consequences. Retrieval generalizes by similarity to stored cases; weight-based memory generalizes by applying abstract rules to inputs the agent has never encountered. Agents built exclusively on retrieval accumulate notes without ever developing expertise, face a provable generalization ceiling on compositionally novel tasks that no increase in context window size can overcome, and structurally convert transient prompt injections into persistent compromise. A Reflexion agent that accumulates thousands of verbal self-critiques is still running the same frozen model at every session; its filing cabinet grows while its capacity does not ( 37 ) . We argue that this conflation is a category error: treating a memo as a mind. Cognitive science distinguishes exemplar-based cognition (generalizing by similarity to stored cases) from rule-based cognition (applying abstract principles extracted from, but no longer dependent on, those cases) ( 29 ; 2 ) . The transition between the two is, in the brain, the consolidation of hippocampal episodes into neocortical weights during sleep ( 26 ; 31 ) . 8 showed that physics novices categorize problems by surface features (“inclined plane problems”) while experts categorize by deep structural principles (“conservation of energy problems”)—a reorganization that is not a change in what they have stored , but in how knowledge is represented . Current AI agents implement only the hippocampal half; no consolidation path to the neocortical half exists in any deployed system. The stakes are not merely philosophical. Developers who deploy long-running agents under the assumption that accumulated retrieval constitutes growing expertise will find that capability stagnates while attack surface grows. Benchmark designers who evaluate agents on seen task distributions will miss the generalization gap that opens on genuinely novel inputs. The continual learning community, whose methods could close this gap, has largely ceded the agentic setting to retrieval-based approaches ( 17 ) . None of these communities can course-correct without first recognizing that retrieval and memory are not interchangeable. We advance four claims. First (Definitional) , agentic memory is exemplar-based lookup: it cannot extrapolate to compositionally novel situations that no stored case covers. Second (Structural) , we prove a Generalization Gap theorem showing retrieval-based memory has a provably lower ceiling than weight-based memory on novel tasks, independently of context window size or retrieval quality. Third (Dynamic) , agents operating exclusively via C -engineering cannot develop expertise; each session begins from the same frozen weights. Fourth (Security) , agentic memory structurally converts transient prompt injection into persistent compromise. We close with a co-existence architecture and a call to action for system builders, benchmark designers, and the continual learning community. We use exemplar-based , lookup , and memo interchangeably for retrieval-based systems, and rule-based , function , and true memory for weight-based systems; Table 1 formalizes the distinction. 2 The Case: Agentic Memory Is a Memo, Not True Memory Table 1: Memory taxonomy for LLM agents. All current agentic systems occupy the “Episodic” row. The “Experiential” row (weight-based encoding of the agent’s lived experience) is the gap. Type Substrate Persists Updated by Generalizes Working Context window Session only Token generation Limited by L Episodic External store Cross-session Read/write ops Exemplar-based Semantic Model weights Permanent Pre-training Rule-based Experiential Model weights Permanent Fine-tuning/CL Rule-based Table 1 maps the landscape. The “Experiential” row is systemically absent from all deployed systems ( 15 ) . This absence is not accidental: it reflects a field-wide assumption that agentic memory (the “Episodic” row) is a sufficient substitute. We argue it is not. A unifying lens: two structurally distinct paths. As an analytical lens, every technique that changes what an LLM agent outputs can be characterized as belonging to one of two structurally distinct categories (acknowledging that some mechanisms, such as persistent test-time adapters, sit at the boundary—we return to this in Alternative 5): (1) Change \theta : modify the model’s weights via pre-training, fine-tuning, reinforcement learning, or any other gradient-based update, changing the prior distribution P(X\mid\theta) ; (2) Change C : inject content into the context window via prompting, RAG, MCP tool calls, skill files, scratchpads, or any other form of context engineering, conditioning generation on P(X\mid\theta,C) . These two paths are structurally different , not merely different in implementation. Changing \theta compresses knowledge into the model’s weight space, whose capacity scales with the number of parameters. Changing C compresses knowledge into text, whose capacity is bounded by the context window length L . The critical asymmetry is not compression ratio but generativity : \theta -compression is generative (the model can recombine weight-encoded rules to handle inputs it has never seen); C -compression is retrieval-based (the model can only use what is explicitly present in context). All current deployed agentic memory is C -engineering. Every technique in Table 1 ’s “Episodic” row (MemGPT, RAG, Reflexion, Voyager) changes C , not \theta . The weights are identical before and after the agent’s experience. The Experience Compression Spectrum. A recent line of work ( 51 ) formalizes memory, skills, and rules as lying on a single experience compression spectrum , differing only in their compression ratio: raw traces (low compression, high fidelity) \to natural-language skills (medium compression,