ENGRAM: Effective, Lightweight Memory Orchestration for Conversational Agents
Daivik Patel, Shrenik Patel, Rutgers University · 2025
Read ENGRAM: Effective, Lightweight Memory Orchestration for Conversational Agents by Daivik Patel, Shrenik Patel, Rutgers University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12960.txt.
Introduction
Daivik Patel and Shrenik Patel present ENGRAM, a deliberately simple memory system for conversational agents. Chatbots in production must remember prior sessions, respect user preferences, and ground answers in past events. Existing systems often pile up knowledge graphs and schedulers, hard to build and reproduce. ENGRAM instead stores each conversation turn as a typed record in one of three memory kinds: episodic, semantic, and procedural, all handled by a single router and retriever.
At query time it fetches the nearest records of each type and merges them as context. On LoCoMo, a long-horizon conversational benchmark, it reached state-of-the-art results. On LongMemEval it beat the full-context baseline by fifteen points while using about one percent of the tokens. Careful memory typing plus plain dense retrieval, they claim, does the job of elaborate architectures at a fraction of the cost.
Catalog record
| CREATOR | Daivik Patel, Shrenik Patel, Rutgers University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12960
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ENGRAM: Effective, Lightweight Memory Orchestration for Conversational Agents Daivik Patel, Shrenik Patel, Rutgers University Abstract Large language models (LLMs) deployed in user-facing applications require long-horizon consistency: the capacity to remember prior interactions, respect user preferences, and ground reasoning in past events. However, contemporary memory systems often adopt complex architectures such as knowledge graphs, multi-stage retrieval, and operating-system–style schedulers, which introduce engineering complexity and reproducibility challenges. We present ENGRAM , a lightweight state-of-the-art memory system that organizes conversation into three canonical memory types—episodic, semantic, and procedural—through a single router and retriever. Each user turn is converted into typed memory records with normalized schemas and embeddings and persisted in a database. At query time, the system retrieves top-k dense neighbors per type, merges results with simple set operations, and provides relevant evidence as context to the model. ENGRAM attains state-of-the-art results on the LoCoMo benchmark, a realistic multi-session conversational question-answering (QA) suite for long-horizon memory, and exceeds the full-context baseline by 15 absolute points on LongMemEval, an extended-horizon conversational benchmark, while using only ≈ 1 % {\approx}1\% of the tokens. Our results suggest that careful memory typing and straightforward dense retrieval enable effective long-term memory management in language models, challenging the trend toward architectural complexity in this domain. 1 Introduction LLMs are now embedded in personal assistants, tutoring systems, productivity tools and many other user-facing applications. These deployments demand long-horizon consistency, meaning remembering prior interactions, respecting user preferences, and grounding reasoning in past events ( Baddeley & Hitch 1974 ) . However, unlike humans, LLMs reset once input falls outside the context window. This leads to brittle behaviors such as forgetting, contradictions, or reliance on pre-training ( Liu et al. 2023 ) . Prior efforts extended the Transformer architecture ( Vaswani et al. 2017 ) to increase effective context, but they do not obviate the need for external memory ( Dai et al. 2019 ; Beltagy et al. 2020 ) . Contemporary memory systems have converged on increasingly elaborate architectures, often involving knowledge graphs, multi-stage retrieval pipelines, or operating-system style schedulers. These designs introduce engineering complexity and many degrees of freedom, making reproducibility and analysis difficult. In contrast, we argue for a different point on the design spectrum: a compact memory layer that is intentionally simple yet sufficient to deliver state-of-the-art accuracy and reliable performance. We introduce ENGRAM , a lightweight memory system that separates three memory types — episodic , semantic , and procedural — and combines them through a single router and a single retriever. Each user message is converted into typed memory records with normalized JSON schemas and embeddings. Records are persisted in a local SQLite store, and at query time the system retrieves top-k neighbors from each memory type, merges results, and provides the evidence set as context to the answering prompt ( Guu et al. 2020 ) . Our central claim is that careful memory typing, minimal routing, and straightforward retrieval suffice to achieve state-of-the-art performance on LoCoMo, and to surpass full-context baselines on LongMemEval ( Maharana et al. 2024 ; Wu et al. 2025 ) . We demonstrate consistent gains across single-hop, multi-hop, open-domain, and temporal categories. Beyond headline metrics, the simplicity of ENGRAM makes it an attractive foundation for principled experimentation: each component is small, interpretable, and easy to swap out. Finally, we provide a complete implementation and evaluation harness to support rigorous comparison and foster adoption by the community. 2 Related Work Existing work on long-term memory for language models spans non-parametric retrieval methods ( Khandelwal et al. 2020 ) , graph-based structures, and system-level abstractions. Non-parametric approaches augment a model with an external store accessed through dense or lexical retrieval ( Lewis et al. 2020 ; Guu et al. 2020 ) . Nearest-neighbor LMs and large-scale retrieval-pretraining extend this line ( Borgeaud et al. 2022 ) . They are attractive for freshness and editability, yet often depend on retriever calibration and heuristic chunking. Graph-based methods organize memories as nodes and relations to support structured traversal and multi-hop reasoning ( Anokhin et al. 2024 ; Li et al. 2024 ) . These designs capture compositional structure but introduce orchestration complexity and latency overhead at inference time. System-level approaches treat memory as a schedulable resource, adding lifecycle management and governance primitives ( Packer et al. 2023 ; Li et al. 2025 ) . Our work is closest to recent memory modules aimed at practical deployment. In contrast to multi-layer schedulers and heavy graph construction, ENGRAM retains the benefits of typed memory and semantic retrieval while minimizing moving parts , yielding a system that is straightforward to operate in both research and production at scale. 3 The ENGRAM Architecture Figure 1: System overview of ENGRAM . Turns are routed into typed stores, embedded, persisted, and later retrieved with semantic search before being passed as context to an answering model. The diagram highlights both the memory creation stage (routing and extraction) and the retrieval stage (top- k selection, aggregation, and prompt injection). Numbers (1)–(5) mark the main components and are referenced below. A router (1) determines which memory buckets apply to an incoming utterance. For each selected bucket, a lightweight extractor converts the utterance into a normalized record and requests an embedding (2). Records are persisted in SQLite together with their embeddings. At query time, the system retrieves the top- k items by cosine similarity from all buckets (3), merges and deduplicates the results (4), and supplies the selected snippets as context to the answering model (5). The stages (1)–(5) together with the overall architecture are shown in Figure 1 . We formalize each stage in the subsections that follow, and provide an end-to-end QA walkthrough in Appendix C . 3.1 Problem Setup We model a dialogue as a sequence of turns \mathcal{C}=\{x_{t}\}_{t=1}^{T},\quad x_{t}=(s_{t},u_{t},\tau_{t}) where s_{t}\in\{A,B\} denotes the speaker identity, u_{t}\in\mathcal{U} is the turn text drawn from the space of natural language, and \tau_{t}\in\mathbb{R}_{+} is a temporal marker. ENGRAM learns a mapping f:\mathcal{C}\mapsto\mathcal{M} that transforms a dialogue \mathcal{C} into a durable memory state \mathcal{M} capable of supporting answers to questions q\in\mathcal{U} posed after the conversation has unfolded. 3.2 Routing and Storage Every turn must be mapped into one or more memory types. ENGRAM employs a router (1) that decides which of the three stores a turn u_{t} should update, represented as a compact three-bit mask b_{t} r(u_{t})\in\{0,1\}^{3}\;\;\Rightarrow\;\;b_{t}=\big(b^{\mathrm{epi}}_{t},b^{\mathrm{sem}}_{t},b^{\mathrm{pro}}_{t}\big) When the router outputs a one for a given type k\in\{\mathrm{epi},\mathrm{sem},\mathrm{pro}\} , the turn is written to that store. For each selected type k , the system creates a structured record from the turn and pairs it with an embedding vector e\in\mathbb{R}^{d} produced by the encoder g:\mathcal{U}\to\mathbb{R}^{d} . Each memory record therefore has two parts: an interpretable set of fields (e.g., text and timestamp) and a dense vector representation suitable for similarity search. Because the router produces a compact three-bit mask, the routing process remains both interpretable and easy to ablate. 3.3 Typed Memory Stores Once turns are routed and stored, ENGRAM organizes them into three typed stores (2): episodic m^{\mathrm{epi}} , semantic m^{\mathrm{sem}} , and procedural m^{\mathrm{pro}} . Episodic memory encodes events that unfold in time, semantic memory preserves stable facts or preferences, and procedural memory retains instructions or workflows ( Tulving 1972 ; Cohen & Squire 1980 )