Evaluating Long-Term Memory for Long-Context Question Answering
Alessandra Terranova, Björn Ross, Alexandra Birch · 2025
Read Evaluating Long-Term Memory for Long-Context Question Answering by Alessandra Terranova, Björn Ross, Alexandra Birch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23730.txt.
Introduction
Alessandra Terranova and two co-authors compare ways of giving large language models a memory for long conversations. True conversational continuity needs memory, and many designs compete: retrieval-based semantic memory, episodic memory from in-context learning, agentic memory, and prompt tuning as procedural memory. Which kind helps most, and for which model, was unclear.
The team tested these approaches on long dialogues annotated with questions that demand varied reasoning. Memory-augmented methods cut token use by over ninety percent while holding accuracy near the full-context level. The best design depended on the model: foundation models gained most from retrieval, while stronger instruction-tuned models gained from episodic reflection and richer agentic memory. One notable finding is that episodic memory helps models recognize the limits of their own knowledge. The lesson is to scale memory complexity with model capability, not to bolt on the fanciest system by default.
Catalog record
| CREATOR | Alessandra Terranova, Björn Ross, Alexandra Birch |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23730
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Long-Term Memory for Long-Context Question Answering Alessandra Terranova, Björn Ross, Alexandra Birch Abstract In order for large language models to achieve true conversational continuity and benefit from experiential learning, they need memory. While research has focused on the development of complex memory systems, it remains unclear which types of memory are most effective for long-context conversational tasks. We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks that require diverse reasoning strategies. We analyse full-context prompting, semantic memory through retrieval-augmented generation and agentic memory, episodic memory through in-context learning, and procedural memory through prompt optimization. Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy. Memory architecture complexity should scale with model capability, with small foundation models benefitting most from RAG, and strong instruction-tuned reasoning model gaining from episodic learning through reflections and more complex agentic semantic memory. In particular, episodic memory can help LLMs recognise the limits of their own knowledge. 1 Introduction Memory is at the core of how humans think, learn, and make decisions. Similarly, for large language model (LLM) based agents, memory is fundamental to carry out extended conversations, reason over longer time frames, learn from experience, and act in a coherent way Liu et al. 2025 . LLMs often struggle to maintain coherence and accuracy in long, multi-turn interactions and very extended contexts. They also lack reliable mechanisms for metacognition, for instance they struggle with recognising the limits of their own knowledge or "knowing what they don’t know" Johnson et al. 2024 . These challenges gain more importance as context windows become larger, conversations can span hundreds of turns, and learning from experience becomes essential in real-world applications. Simply feeding models longer context is not always practical or efficient, as it increases inference cost and does not address the need for models to also learn from past information. Techniques such as retrieval-augmented generation (RAG) Lewis et al. 2020 address context limitations and allow updating and controlling data sources to improve the factuality of generated text. Additionally, recent advancements have introduced mechanisms to augment LLMs with memory components, to enhance their ability of retaining information over long contexts, and of learning from experience Shinn et al. 2023 ; Zhao et al. 2024 . In this paper, we examine different memory architectures, semantic (RAG and A-Mem), episodic (in-context learning with reflections), and procedural (prompt optimization), within a unified evaluation framework. We focus on long-context conversational QA, and reflect on how memory augmentation can support learning from experience and improve the models’ ability to reason about their own knowledge. Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy. The complexity of the memory architecture should scale with the model’s capabilities: smaller foundation models benefit most from RAG, while more advanced instruction-tuned models from episodic memory and richer semantic memory structures. Additionally, episodic memory plays an important role in mitigating LLMs’ metacognitive limitations, helping them recognise the limits of their knowledge. This comparison establishes baselines and identifies trade-offs between performance, interpretability, and resource constraints that can inform memory system design. 2 Background 2.1 Memory for LLM agents The need for a dedicated memory component in LLM-based agents seems fundamental to capture the model’s internal reasoning, its task-specific context, historical dialogue, and evolving objectives Liu et al. 2025 ; Zhang et al. 2024 . Similarly to human memory, memory modules for LLMs can be categorised into long and short-term memory. Short-term memory – usually the immediate context window of the LLM – is sufficient to deal with simple tasks, but long-term memory becomes essential in more complex and realistic interaction scenarios Khosla et al. 2023 . Following the organization typically used for human memory, long-term memory is organised into three different mechanisms: semantic memory stores general world knowledge and facts, making it possible for the model to provide informed responses; procedural memory encodes knowledge about how to perform the task at hand; episodic memory finally captures specific past interactions or experiences, allowing the model to gather information that can inform its future decision making Hatalis et al. 2023 . Early approaches to memory through appending conversation history to the input prompt have evolved Liu et al. 2025 , leading to the use of vector embeddings for retrieving memories and selective incorporation of reasoning steps into subsequent inference calls Liu et al. 2023a . A growing body of work has been investigating dedicated memory systems for LLMs, exploring memory storage and management Zhong et al. 2024 ; Roucher et al. 2025 . Some systems maintain logs of each agent interaction Zhong et al. 2024 , while others employ read-write mechanisms that allow models to update memory content as needed Modarressi et al. 2023 . Recent research has been exploring more dynamic and agent-driven approaches to memory. A-mem Xu et al. 2025 proposes a semantic memory system where each memory is structured with contextual tags and dynamically linked to related memories. This enables the system to form an evolving network of knowledge that supports updates over time and memory reorganization. 2.2 Retrieval Augmented Generation Retrieval Augmented Generation (RAG) Lewis et al. 2020 can improve LLMs efficiency and traceability, providing a way to capture and update knowledge in a modular and more interpretable way Guu et al. 2020 . Retrieval-augmented LMs can be more reliable, adaptable, and attributable thanks to the use of large-scale datastores for knowledge-intensive tasks Asai et al. 2024 . The standard RAG process involves chunking a set of documents, embedding and indexing them in a vectorstore, and retrieving relevant chunks based on semantic similarity with the query; this retrieved context can then be used to augment the LLM’s prompt for generation. Recent research has made these systems agentic by introducing more autonomy in determining what to retrieve and when, possibly having different retrievers available and refining the search strategies based on intermediate results Shao et al. 2023 . 2.3 Learning through memory A growing interest has developed in leveraging LLM-based agents’ experience to improve them without relying on costly fine-tuning or parametric updates. Traditional reinforcement learning approaches, while effective in many domains, often require extensive interaction data and prolonged training times. In response to these limitations, several frameworks have been proposed that enable language agents to learn through memory and linguistic feedback. One such approach is Reflexion Shinn et al. 2023 , which allows agents to self-improve through natural language reflection stored in an episodic memory buffer, rather than weight updates. Building on Reflexion, the Experiential Learning (ExpeL) framework Zhao et al. 2024 proposes a memory-based strategy in which agents autonomously learn from experience without gradient updates, highlighting their capacity for transfer learning and emergent reasoning abilities. Our approach to episodic memory, motivated by this literature, also makes use of textual feedback signals and knowledge from past experiences, but it is built with conversational tasks in mind, and to be modular and easy to evaluate in conjunction with different LLM memory components. These developments point to a paradigm shift in how learning is implemented in LLM-based agents, moving towards dynamic, introspective, and context-aware memory systems. 3 Methodologies 3.1 Dataset and evaluation Our goal is to