EngramaBench : Evaluating Long-Term Conversational Memory with Structured Graph Retrieval

Julián Acuña · 2026

Read EngramaBench : Evaluating Long-Term Conversational Memory with Structured Graph Retrieval by Julián Acuña (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21229.txt.

Introduction

Julian Acuna introduces EngramaBench, a benchmark for long-term conversational memory built from five personas, one hundred multi-session chats, and one hundred fifty queries spanning recall, cross-space reasoning, time reasoning, abstention, and synthesis. He tests Engrama, a graph-structured memory system, against full-context prompting and Mem0 with the same answering model. Full-context prompting wins overall, but Engrama alone beats it on cross-space reasoning, while Mem0 is cheapest and weakest.

Catalog record

CREATORJulián Acuña
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29562
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21229

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EngramaBench : Evaluating Long-Term Conversational Memory with Structured Graph Retrieval Julián Acuña Abstract Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce EngramaBench , a benchmark for long-term conversational memory built around five personas, one hundred multi-session conversations, and one hundred fifty queries spanning factual recall, cross-space integration, temporal reasoning, adversarial abstention, and emergent synthesis. We evaluate Engrama , a graph-structured memory system, against GPT-4o full-context prompting and Mem0 , an open-source vector-retrieval memory system. All three use the same answering model (GPT-4o), isolating the effect of memory architecture. GPT-4o full-context achieves the highest composite score (0.6186), while Engrama scores 0.5367 globally but is the only system to score higher than full-context prompting on cross_space reasoning (0.6532 vs. 0.6291, n = 30 n{=}30 ). Mem0 is cheapest but substantially weaker (0.4809). Ablations reveal that the components driving Engrama ’s cross-space advantage trade off against global composite score, exposing a systems-level tension between structured memory specialization and aggregate optimization. 1 Introduction Large language model assistants are increasingly expected to behave as if they remember. Users interact with them as persistent collaborators that accumulate context over weeks and months, requiring the system to retain, recover, update, and connect information from prior interactions. These are fundamentally long-term memory problems. One natural approach is to provide the model with as much prior history as possible at answer time. Recent long-context models make this strategy viable, and it remains surprisingly strong. However, full-context prompting scales poorly in cost and treats memory as transcript inclusion rather than structured representation. This leaves open whether it is the right long-term solution. This tension motivates two questions. First, what kinds of memory behavior actually matter in long-term conversational systems? Second, when does structured memory provide an advantage over brute-force context inclusion or flat retrieval? Existing benchmarks partially address these questions but often omit cross-space integration, temporal change, or abstention under multi-session interaction. Many memory systems are also difficult to analyze because they conflate memory construction, retrieval, and answer generation. We present two contributions. First, we introduce EngramaBench , a benchmark for long-term conversational memory built around five canonical personas, one hundred multi-session conversations, and one hundred fifty queries spanning factual recall ( single_space ), cross-space integration ( cross_space ), temporal reasoning ( temporal_cross_space ), adversarial abstention ( adversarial ), and emergent synthesis ( emergent_insight ). Second, we evaluate Engrama , a graph-structured memory system, against GPT-4o full-context and Mem0 ( Chhikara et al. 2025 ) , an open-source vector-retrieval memory system. The resulting picture is nuanced. Full-context prompting achieves the highest composite score ( 0.6186 ), but Engrama is the only system to score higher on cross_space reasoning ( 0.6532 vs. 0.6291 ; n{=}30 ), while Mem0 trails both ( 0.4809 ). Ablations reveal that the components driving Engrama ’s cross-space strength trade off against aggregate score, exposing a design tension that helps explain why memory architectures can appear promising in smaller evaluations but reveal costs under broader benchmarking. Our contributions: • EngramaBench , a benchmark with explicit coverage of five memory behavior families including cross-space reasoning as a first-class target. • A controlled comparison showing that full-context prompting wins globally while Engrama scores highest on cross_space at substantially lower query-time cost. • Ablation analysis separating the mechanisms that improve cross-space behavior from those that maximize aggregate score. The benchmark data, scorer, and canonical results are publicly available. 1 1 1 Publicly available at: https://github.com/julianacunadc/engramabench 2 Related Work Memory-augmented assistants. One strong baseline for long-term conversational tasks is full-context prompting: providing the model with the entire prior history at answer time. This avoids deciding what to remember but scales poorly and offers little control over memory organization. This limitation motivates memory-augmented systems that explicitly index, compress, or retrieve information from prior interactions, including MemoryBank ( Zhong et al. 2023 ) , MemGPT/Letta ( Packer et al. 2023 ) , ReadAgent ( Lee et al. 2024 ) , PerLTQA ( Du et al. 2024 ) , Mem0 ( Chhikara et al. 2025 ) , and LongMemEval ( Wu et al. 2025 ) . Graph-structured retrieval systems such as GraphRAG ( Edge et al. 2024 ) and HippoRAG ( Gutiérrez et al. 2024 ) explore how structured substrates can support multi-hop recall more effectively than flat retrieval. Engrama is closest in spirit to this line but differs in treating long-term conversational state as a graph-structured memory surface organized around entities, spaces, temporal traces, and cross-space associations. Benchmarks for conversational memory. Early multi-session benchmarks such as MSC ( Xu et al. 2022 ) emphasize coherence and persona consistency over limited sessions. LoCoMo ( Maharana et al. 2024 ) introduces very long multi-session conversations with temporal event graphs, showing that even strong systems struggle as histories grow. LongMemEval ( Wu et al. 2025 ) focuses on chat assistants and decomposes memory into indexing, retrieval, and reading stages. However, prior benchmarks often underemphasize: (i) explicit integration across semantically distinct parts of a user’s life, (ii) the interaction between temporal change and cross-session reasoning, (iii) abstention under fabricated queries, and (iv) synthesis whose answers are not stated verbatim in any session. Positioning. EngramaBench complements this prior work by making cross-space reasoning a first-class benchmark target alongside adversarial abstention and emergent synthesis under controlled persona histories. It trades breadth for interpretability: by organizing each persona around recurring semantic spaces, it distinguishes memory behaviors that are often conflated under a single long-context score. 3 EngramaBench We introduce EngramaBench , a benchmark for evaluating long-term conversational memory under multi-session interaction. It tests not only isolated fact recovery but also cross-space integration, temporal reasoning, abstention under fabricated queries, and emergent synthesis from dispersed evidence. 3.1 Problem Formulation Each benchmark instance is a tuple (H,q,a,E) : a chronologically ordered interaction history H , a query q posed after that history, a reference answer a , and annotated evidence E . The full benchmark contains 5 personas , 100 conversations , and 150 queries . Each persona contributes 20 conversations and 30 queries. Queries are posed only after the full history has been observed, so the task is delayed long-term recall rather than next-turn prediction. 3.2 Dataset Construction The benchmark is organized around five canonical personas: Priya Sharma (PI/professor), Carlos Mendoza (startup founder), Sam Torres (artist/musician), Diane Chen (product lead), and Kai Nakamura (PhD student). Each persona is defined over five recurring semantic spaces that partition the major domains of their life. This space structure creates a memory landscape where some questions are resolvable within a single region while others require combining evidence across regions. The personas span different professional domains, life stages, and discourse styles, reducing overfitting to a single archetype. For each persona, 20 timestamped conversations trace a medium-horizon narrative (late 2025 to early 2026). Conversations are synthetically authored through an LLM-assisted generation pipeline constrained by persona blueprints, cross-space threads, and temporal arcs, then checked for structural consistency. Query generation follows a fixed per-persona distribution: 6 single_space , 6 cross_space , 6 temporal_cross_space , 4 emergent_insight , and 8 adversarial queries (30 total per persona). 3.3 Query Taxonomy EngramaBench evaluates five task families: • single_space : factual recall within a single semantic space. • cross_space : integration across two

Read the complete text