E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memory

2026

Read E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memory (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21714.txt.

Introduction

A memory system for language model agents rebuilds past episodes in full context instead of compressing them. Long, careful problem solving needs memory that stays faithful over long spans. Most current memory tools squeeze past events into embeddings or graphs before storing them.

That squeezing cuts the context that gives each event its meaning. The authors' multi-agent design reconstructs episodes with their sequential links intact. Keeping context whole should support deeper reasoning across long tasks. The work targets agents that must stay coherent over hours of work, where a mangled memory can quietly wreck the whole chain.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21714

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memory Abstract The evolution of Large Language Model (LLM) agents towards System 2 reasoning, characterized by deliberative, high-precision problem-solving, requires maintaining rigorous logical integrity over extended horizons. However, prevalent memory preprocessing paradigms suffer from destructive de-contextualization. By compressing complex sequential dependencies into pre-defined structures (e.g., embeddings or graphs), these methods sever the contextual integrity essential for deep reasoning. To address this, we propose E-mem, a framework shifting from Memory Preprocessing to Episodic Context Reconstruction. Inspired by biological engrams, E-mem employs a heterogeneous hierarchical architecture where multiple assistant agents maintain uncompressed memory contexts, while a central master agent orchestrates global planning. Unlike passive retrieval, our mechanism empowers assistants to locally reason within activated segments, extracting context-aware evidence before aggregation. Evaluations on the LoCoMo benchmark demonstrate that E-mem achieves over 54% F1, surpassing the state-of-the-art GAM by 7.75%, while reducing token cost by over 70%. Kaixiang Wang 1∗ , Yidan Lin 1∗ , Jiong Lou 1 , Zhaojiacheng Zhou 1 , Bunyod Suvonov 1 , Jie Li 1 1 Shanghai Jiao Tong University, China 1 Introduction Large Language Models (LLMs) have evolved from stochastic text generators into the central cognitive controllers of autonomous agents ( 27 ; 25 ; 16 ) . Empowered by planning capabilities and external tool integration ( 15 ) , these systems are now transitioning towards System 2 reasoning—characterized by deliberative, sequential problem-solving in dynamic environments ( 34 ; 31 ; 20 ) . However, this shift to high-precision cognition demands strict adherence to rigorous causal chains. In these scenarios, maintaining extensive history ( 35 ) becomes pivotal to preserve the logical integrity essential for deep, long-horizon planning ( 18 ; 24 ; 21 ) . Figure 1 : Traditional Memory System vs. E-mem Nevertheless, expanding operational horizons presents significant memory challenges. Merely extending context windows often triggers the “Lost-in-the-Middle” phenomenon ( 13 ) , thereby necessitating robust memory management mechanisms ( 33 ; 17 ) . Existing approaches primarily rely on preprocessing to index memory, mapping raw, unstructured contexts into fixed, pre-defined structures (e.g., static embeddings, knowledge graphs, or hierarchical archives) ( 28 ; 17 ; 7 ) . While enabling efficient lookup, this strategy results in destructive de-contextualization: by compressing complex sequential dependencies into rigid representations, it disrupts the critical integrity required for deep reasoning ( 19 ; 9 ) . Consequently, such methods struggle to reconstruct complex causal chains or comprehend memories within their original sequential contexts, ultimately yielding suboptimal performance on information-dense benchmarks like LoCoMo ( 14 ) . To address the critical limitations of destructive de-contextualization, particularly in scenarios demanding rigorous logical deduction over memory, we introduce E-mem. This framework transitions from Memory Preprocessing to Episodic Context Reconstruction (Figure 1 ). Inspired by biological engrams, E-mem preserves the full Episodic Context of original experiences. To mitigate the information distortion inherent in ultra-long contexts while ensuring cost-effectiveness for scalable deployment, E-mem adopts a heterogeneous hierarchical Master-Assistant architecture, wherein a central master agent orchestrates global planning, while multiple assistant agents (implemented as small language models) serve as memory nodes. Each assistant agent maintains the raw memory context of a specific segment. For each query, a routing mechanism selectively activates a relevant subset of assistant agents. Crucially, instead of merely retrieving text chunks, these agents execute Episodic Context Reconstruction—a process where they actively ’re-experience’ and reason within the restored native contexts to derive precise, local evidence for the master agent. This mechanism not only mitigates context window constraints and reduces computational costs but also ensures high-fidelity, context-preserving inference, offering a distinct advantage in complex, multi-hop reasoning tasks where traditional retrieval mechanisms often fall short. Empirical evaluations on LoCoMo ( 14 ) and HotpotQA ( 30 ) confirm that E-mem achieves state-of-the-art performance. The system delivers superior accuracy compared to SOTA baselines while significantly reducing token cost, validating the efficiency of the proposed Episodic Context Reconstruction paradigm. In summary, our main contributions are as follows: • Episodic Context Reconstruction. To address the destructive de-contextualization inherent in traditional memory preprocessing, we introduce E-mem, a framework that centered on Episodic Context Reconstruction. Unlike static retrieval methods that sever sequential dependencies, our approach delegates active reasoning to assistant agents. They preserve and process the full context of memory segments locally, ensuring that only logically deduced evidence—rather than raw, noisy fragments—is surfaced to master agents. • Heterogeneous Hierarchical Master-Assistant Architecture. We propose E-mem, a scalable framework that decouples high-level planning from memory retention. By coordinating a master agent with lightweight, SLM-based assistant agents, our design mitigates the information loss inherent in preprocessing, ensuring high-fidelity reasoning across long horizons without suffering from the “lost-in-the-middle” phenomenon. • SOTA Performance with Token Efficiency. Extensive evaluations on LoCoMo and HotpotQA demonstrate that E-mem outperforms strong baselines by an average of 7.75% (F1), with notable gains in complex multi-hop (+8.56%) and temporal reasoning (+8.87%). Crucially, these improvements are realized while reducing token cost by over 70%. These results confirm E-mem as a vital complement to traditional memory paradigms for System 2 reasoning. 2 Related Work Figure 2 : Overview of E-mem Retrieval-Augmented Generation. RAG has established itself as a fundamental paradigm to mitigate LLM hallucination and knowledge obsolescence by grounding generation in external corpora ( 12 ; 11 ; 6 ) . While standard frameworks typically employ a vector-based “retrieve-then-generate” pipeline, recent Agentic RAG grants systems greater autonomy in retrieval planning and iterative context refinement ( 8 ; 2 ; 22 ; 31 ) . Despite these advancements in retrieval logic, the underlying storage mechanism remains predominantly reliant on preprocessing. This approach inherently compresses rich sequential contexts into fixed geometric points, risking the loss of fine-grained sequential dependencies essential for reasoning ( 13 ; 26 ; 3 ) . Memory Systems for Autonomous Agents. Beyond simple retrieval, persistent and adaptive memory systems are critical for agents to maintain long-term interaction coherence. Addressing the finite context window, MemGPT ( 17 ) employs an operating system-inspired virtual context management technique via hierarchical paging. However, this paradigm relies on swapping fragmented chunks, necessitating redundant re-processing to restore sequential dependencies. Other works focus on optimizing memory structure and active evolution. For instance, G-Memory ( 32 ) introduces a graph-based hierarchical structure to enable navigation between global macro-views and local micro-interactions. Similarly, A-Mem ( 28 ) adopts a self-evolving framework based on the Zettelkasten method, while GAM ( 29 ) and ReasoningBank leverage multi-agent deep research and reasoning trajectory storage, respectively. Although recent efforts extend to personalization (Mem0 ( 4 ) ) and benchmarking (MemoryBench ( 1 ) ), these approaches remain bound to text-based preprocessing paradigms. By compressing complex contexts into rigid structures, they often disrupt the contextual integrity required for deep reasoning. In contrast, E-mem introduces Episodic Context Reconstruction , directly preserving full memory contexts. This approach ensures seamless inference integrity, serving as a critical complement to existing paradigms for high-precision, complex reasoning tasks. 3 Method Cognitive science defines memory as the re-experience of intact episodic contexts rather than static retrieval ( 23 ) . In contrast, prevalent preprocessing paradigms force dynamic inputs into fixed structures, resulting in destructive de-contextualization. This rigid compression disrupts sequential dependencies, severing the contextual integrity essential for deep reasoning. We propose E-mem, a framework centered

Read the complete text