EverMemBench: Benchmarking Long-Term Interactive Memory in Large Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Information extraction Chuanrui Hu, Tong Li, Xingze Gao, Hongda Chen, Yi Bai, Dannong Xu, Tianwei Lin, Xinda Zhao, Xiaohong Li, Yunyun Han, Jian Pei, Yafeng Deng Abstract . Long-term conversational memory is essential for LLM-based assistants, yet existing benchmarks focus on dyadic, single-topic dialogues that fail to capture real-world complexity. We introduce EverMemBench , a benchmark featuring multi-party, multi-group conversations spanning over 1 million tokens with temporally evolving information, cross-topic interleaving, and role-specific personas. EverMemBench evaluates memory systems across three dimensions through 1,000+ QA pairs: fine-grained recall , memory awareness , and user profile understanding . Our evaluation reveals critical limitations: (1) multi-hop reasoning collapses in multi-party settings, with even oracle models achieving only 26%; (2) temporal reasoning remains unsolved, requiring version semantics beyond timestamp matching; (3) memory awareness is bottlenecked by retrieval, where current similarity-based methods fail to bridge the semantic gap between queries and implicitly relevant memories. EverMemBench provides a challenging testbed for developing next-generation memory architectures. 1. Introduction Large language models are increasingly used as conversational agents in settings where interactions extend over time and across contexts ( 3 ; 13 ) . In realistic scenarios such as workplace collaboration and personal assistance, conversations are often multi-party, topics are interleaved and revisited, and previously stated information is frequently revised. A memory-capable system in these environments must do more than retrieve a past fact on demand. It must retain fine-grained details, recognize when earlier information becomes relevant in a new situation, and respond in a way that is consistent with user preferences and social context. Figure 1. Existing benchmarks vs. EverMemBench. Prior benchmarks adopt dyadic settings with isolated topic sessions. EverMeBench introduces multi-party collaboration across interdependent groups, where relevant information is scattered across speakers, channels, and time—demanding cross-group reasoning and temporal tracking absent in dyadic scenarios. teaser Although long-context modeling and memory-augmented agents have advanced rapidly, evaluation remains a bottleneck. Many benchmarks treat memory as recall over long inputs and implicitly equate stronger memory with the ability to handle more tokens ( 19 ; 28 ; 14 ; 26 ; 23 ) . However, real failures often come from different sources: confusion about who said what in group chats, interference across closely related topics, inconsistency in persona and style, and inability to update beliefs when plans change. Meanwhile, recent memory-augmented systems ( 5 ; 25 ; 22 ; 18 ; 16 ) increasingly attach external or structured memory to LLMs for long-horizon personalization and task continuity, typically storing user facts and preferences and retrieving and updating them over time. Their rapid adoption makes realistic and diagnostic benchmarks more urgent, since it remains unclear which designs improve behavior under realistic conversational dynamics rather than only improving recall in constructed long contexts. This mismatch between existing benchmarks and realistic memory demands is summarized in Table 1 . A common limitation of existing benchmarks is the reliance on simplified interaction structures ( 11 ) . Many focus on dyadic conversations, while real interactions frequently involve multiple roles contributing to a single decision. Many construct long contexts by adding topic-irrelevant distractors, which tests noise tolerance but does not reflect how real conversations evolve, where topic transitions are usually coherent and interleaved ( 4 ; 23 ) . Persona modeling is often shallow, despite the fact that communication style depends on role, expertise, and social power relations. Finally, most benchmarks assume stationary facts, whereas real user knowledge is non-stationary and requires explicit update and conflict resolution. To address these gaps, we introduce EverMemBench, a memory benchmark designed to better match real conversational dynamics and to evaluate memory beyond surface recall. EverMemBench is built from high information density multi-party dialogues in which multiple roles meaningfully participate. The dialogues exhibit coherent cross-topic interactions, including interleaving and returning to earlier threads, rather than abrupt topic jumps or unrelated padding. The benchmark includes diverse personas with distinct skills and communication styles, and it models style shifts conditioned on social context and role relations. Crucially, it supports dynamic user knowledge updates, where earlier information can be revised as constraints change. The dialogues are additionally grounded with realistic references such as news and calendar-like temporal structure, improving realism of both content and timing. An overview comparison with prior benchmarks is shown in Figure 1 . EverMemBench also provides an evaluation suite that decomposes memory competence into three task families: fine-grained detailed recall, memory awareness, and user profile understanding. Detailed recall tests precise extraction of specific facts. Memory awareness tests whether the system can identify when historical information is relevant and use it appropriately in a new scenario. User profile understanding tests whether the system can infer and maintain consistent personalization aligned with the user and the social setting. Together, these tasks evaluate not only what the system remembers, but also when memory should be activated and how it should shape behavior. Aspects LoCoMo LongMemEval PersonaMem-v1 PersonaMem-v2 EverMemBench (Ours) Dialogue Characteristics Interaction Type Dyadic User–Assistant User–Assistant User–Assistant Multi-party Group Task Structure Single-session Long-term Personalized Preference-based Long-term Interdependent Dialogue Turns 326.8 493.5 313.6 448.5 10,204.6 Context Length 9K 1.5M 1M 128K 1M Personas per Batch 2 1 1 1 37.6 Dialogue Features High-Info Dialog Flow ✗ ✗ ✗ ✗ ✓ Diverse Persona Interaction ✗ ✗ ✗ ✗ ✓ Cross-Topic Interaction ✗ ✗ ✗ ✗ ✓ User Knowledge Update ✗ ✗ ✓ ✓ ✓ Evaluation Dimensions Fine-Grained Recall ✓ ✓ ✓ ✓ ✓ Memory Awareness ✗ ✗ ✓ ✓ ✓ Profile Understanding † ✗ ✗ ✗ ✗ ✓ † Profile Understanding denotes implicit user modeling from long-term dialogue, not explicit profile retrieval. Table 1. Comparison with prior conversational memory benchmarks. EverMemBench uniquely supports multi-party group conversations with long-term interdependent tasks, high information density, and diverse persona interactions. In summary, our contributions are as follows: • We introduce EverMemBench , the first memory benchmark designed for multi-party, multi-group settings, featuring high information density dialogues across 5 topics with each spanning 1 million tokens, coherent cross-topic interactions, role-conditioned personas, and dynamic knowledge updates. • We decompose memory competence into three evaluation dimensions —fine-grained recall, memory awareness, and user profile understanding—moving beyond simple recall to assess when memory should be activated and how it should shape personalized responses. • We conduct systematic evaluation revealing that multi-hop reasoning collapses in multi-party settings, temporal reasoning requires version semantics beyond retrieval, and memory awareness is bottlenecked by semantic gaps in similarity-based retrieval. 2. Related Work Long-Context Conversational Memory Benchmarks. Recent benchmarks evaluate long-horizon conversational memory from complementary angles, including long multi-session interaction, capability-factorized assistant memory, and dynamic personalization. LoCoMo ( 19 ) focuses on very long multi-session conversations with tasks such as question answering and event summarization. LongMemEval ( 28 ) decomposes chat-assistant memory into abilities including information extraction, multi-session and temporal reasoning, knowledge updates, and abstention. PersonaMem ( 14 ) emphasizes dynamic user profiling and personalization over multi-session user–LLM histories. More recent efforts broaden coverage of memory abilities and dialogue settings, including MemBench ( 26 ) , MADial-Bench ( 9 ) , and BEAM ( 27 ) . A common simplifying assumption across many benchmarks is that interactions are dyadic or centered on a single assistant and a single user, which can under-stress realistic collaborative phenomena such as dense multi-party attribution, coherent cross-topic interleaving, persona shifts under social context, and non-stationary user knowledge that must be updated and reconciled. EverMemBench complements these benchmarks by explicitly targeting multi-party group chat with high information density, complex cross-topic interactions, diverse personas shaped by role relations, and dynamic user knowledge updates, together with tasks that separate detailed recall, memory awareness, and user profile understanding. Memory-Augmented Systems and Architectures. Memory-augmented systems increasingly treat memory as an explicit component that can be persisted, structured, retrieved, and updated, with designs ranging from pragmatic memory layers to more autonomous organization and OS-level management. Mem0 ( 5 ) and MemInsight ( 25 ) propose scalable persistent memory layers that extract salient information from conversational histories and retrieve it when needed, reporting gains on long-term memory evaluation such as LoCoMo ( 19 ) . Beyond persistent stores, A-MEM ( 29 ) frames memory as an agentic module that decides what to store and how to use it, while Nemori ( 22 ) proposes self-organizing memory that emphasizes structured organization and evolution. Another line elevates memory to an infrastructure abstraction: MemOS ( 18 ) and MemoryOS ( 16 ) treat memory as an OS-like resource, aiming to unify heterogeneous memories with scheduling and lifecycle management. Many such systems adopt retrieval-augmented generation as a core mechanism for accessing external or long-range information ( 4 ; 2 ; 30 ; 17 ; 8 ; 31 ) , and RAG-style indexing and retrieval often serve as a backbone for conversational memory pipelines. While these systems are typically evaluated in dyadic or single-user settings, EverMemBench offers a multi-party, high-density environment that more directly stresses attribution, cross-topic linkage, persona consistency under social context shifts, and dynamic knowledge update handling, enabling more diagnostic comparisons of memory mechanisms. 3. EverMemBench Figure 2. Data curation pipeline of EverMemBench. Stage 1 instantiates a synthetic organization and generates agent profiles, including stable attributes (e.g., roles and hard skills), context-dependent communication styles, and role-consistent sub-tasks. Stage 2 constructs a globally consistent, calendar-aligned timeline from project plans and dated public news, and incrementally synthesizes long-horizon group-chat dialogue blocks conditioned on profiles and rolling summaries; each block is iteratively validated and repaired via programmatic and semantic checks. Stage 3 generates evidence-grounded QA items targeting fine-grained recall, memory awareness, and user-profile understanding, and filters them through a three-phase quality control procedure: LLM blind testing, answerability/uniqueness verification, and human audit. 3.1. Evaluation Dimensions EverMemBench aims to evaluate LLMs as long-term collaborators in complex, dynamic environments. Unlike prior dyadic benchmarks ( 19 ; 28 ; 14 ; 15 ) , real-world collaboration requires navigating dense multi-party coordination, adapting to evolving constraints, and maintaining role-dependent consistency. We instantiate these challenges in a high-fidelity workplace simulation, as it naturally concentrates the core difficulties of realistic memory. To systematically diagnose these capabilities, we decompose memory competence into three complementary task families: • Fine-grained Recall evaluates direct memory lookups, anchoring queries to specific factual details with definitive answers derived from stored records. Sub-tasks include: Single-hop for direct entity retrieval; Multi-hop for cross-group reasoning across personas or channels; and Temporal for chronological reasoning about version histories. • Memory Awareness tests whether the system can mobilize existing memories to solve problems in unseen situations, differentiating active comprehension from passive storage. Sub-tasks include: Constraint for applying implicit rules from prior conversations; Proactivity for surfacing relevant context when evaluating proposed actions; and Updating for identifying current valid states when earlier decisions have been superseded. • Profile Understanding assesses the ability to maintain distinct user personas by mining implicit habits and long-term traits from dialogue history. Sub-tasks include: Style for matching communication patterns; Skill for determining responses based on professional expertise; and Title for inferring focus areas from organizational role. 3.2. Task Formulation We evaluate memory systems via a streaming multi-group protocol. The benchmark comprises 5 projects spanning diverse domains to assess cross-domain adaptability; each project maintains isolated dialogue history, memory state, and evaluation queries. Within each project, N groups engage in a simulated year of daily conversations. During the ingestion phase , the system receives raw multi-party conversations organized by day and group: on each day d , it observes the daily batch \{M_{d,g}\}_{g=1}^{N} , where M_{d,g} denotes the chronologically ordered multi-party message list of group g on day d . The system must autonomously process these interleaved multi-party exchanges to build and maintain its memory state. After full ingestion, the evaluation phase poses queries against the complete history. Queries take two forms: (1) multiple-choice questions scored by exact option matching, and (2) open-ended questions scored by an LLM judge. Each question targets one of the three evaluation dimensions and is annotated with evidence spans for diagnostic analysis. 3.3. Data Construction Generating coherent long-horizon multi-party dialogues poses three key challenges: (1) context-window limits force truncation of early history; (2) logical inconsistencies emerge as conversations grow; (3) persona and temporal coherence degrade over extended interactions. To address these, we decompose data construction into three stages (Figure 2 ): Stage 1 generates a global blueprint encoding organizational structure, individual persona profiles, and a project timeline with main and auxiliary storylines; Stage 2 synthesizes dialogues in chunks conditioned on the blueprint; Stage 3 generates evidence-grounded QA pairs from the dialogues and blueprint. This separation ensures global consistency while enabling scalable, chunk-wise generation. All generation processes (blueprint, dialogue, and QA) employ Gemini-2.5-Pro ( 6 ) as the backbone model. 3.3.1. Preliminaries We first establish the shared foundation for data generation. We predefine an organizational skeleton S comprising 300 employees \mathcal{E}=\{e_{1},\dots,e_{300}\} across 7 departments, structured in a three-tier hierarchy (1 CEO, 7 directors, 292 staff), and five projects about different domains \mathcal{P}=\{p_{1},\dots,p_{5}\} . Given S , we utilize LLM to generate a persona profile for each employee: (1) \pi_{e}=\big(\,\mathrm{rank}_{e},\;\mathrm{dept}_{e},\;\mathrm{role}_{e},\;\mathbf{s}_{e},\;\mathbf{c}_{e}\,\big), where \mathbf{s}_{e} is a skill set (40–60 competencies) and \mathbf{c}_{e} is an 8-dimensional communication style vector. 3.3.2. Blueprint Generation For each project p\in\mathcal{P} , we construct an independent blueprint B_{p} that guides all subsequent dialogue generation. The blueprint encodes: (2) B_{p}=\big(\,\mathcal{E}_{p},\;\{(\mathcal{E}_{p,j},\,\mathcal{T}_{p,j})\}_{j=1}^{3}\,\big), where \mathcal{E}_{p}\subset\mathcal{E} is the project team (a subset of 20–60 employees selected from the full pool), and for each of the three sub-projects j\in\{1,2,3\} : \mathcal{E}_{p,j}\subseteq\mathcal{E}_{p} is the assigned member set (with possible overlaps across sub-projects), and \mathcal{T}_{p,j} is the set of sub-tasks generated based on member profiles and sub-project goals. Together with the shared foundation (S,\Pi) , the blueprint B_{p} enforces decision-logic consistency, persona consistency, and temporal coherence during dialogue generation. We construct B_{p} through the following steps: (1) select a project team \mathcal{E}_{p} (20–60 members) from the employee pool via skill matching; (2) decompose project p into 3 concurrent sub-projects; (3) assign members to each sub-project \mathcal{E}_{p,j} , where one member may participate in 1–3 sub-projects; (4) generate sub-tasks \mathcal{T}_{p,j} for each sub-project based on member profiles and project requirements. 3.3.3. Conversation Generation Dialogues are generated incrementally for each project p over a simulated timeline of D days. Each sub-project j\in\{1,2,3\} maintains a dedicated group chat; however, since all three serve the same parent project, cross-group information sharing naturally emerges through overlapping membership and inter-dependent sub-tasks. A key challenge is maintaining coherence over hundreds of days while respecting LLM context limits. We address this through hierarchical summarization. Let C_{p,j}^{(