Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents

Natchanon Pollertlam, Witchayut Kornsuwannawit · 2026

Read Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents by Natchanon Pollertlam, Witchayut Kornsuwannawit (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04814.txt.

Introduction

Pollertlam and Kornsuwannawit compare fact-based memory with long-context LLMs for persistent agents. Systems choose between passing full histories to a long-context model and a memory system for structured facts. The study pits Mem0 memory against long-context inference on three benchmarks. Long-context GPT-5-mini wins recall on LongMemEval and LoCoMo, while memory stays close on PersonaMemv2. A cost model with prompt caching shows different cost profiles. Long-context charges grow per turn with context length, while memory read costs stay fixed after one write phase.

Catalog record

CREATORNatchanon Pollertlam, Witchayut Kornsuwannawit
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43057
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04814

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents Natchanon Pollertlam, Witchayut Kornsuwannawit Abstract Persistent conversational AI systems face a choice between passing full conversation histories to a long-context large language model (LLM) and maintaining a dedicated memory system that extracts and retrieves structured facts. We compare a fact-based memory system built on the Mem0 framework against long-context LLM inference on three memory-centric benchmarks—LongMemEval, LoCoMo, and PersonaMem v2—and evaluate both architectures on accuracy and cumulative API cost. Long-context GPT-5-mini achieves higher factual recall on LongMemEval and LoCoMo, while the memory system is competitive on PersonaMem v2, where persona consistency depends on stable, factual attributes suited to flat-typed extraction. We construct a cost model that incorporates prompt caching and show that the two architectures have structurally different cost profiles: long-context inference incurs a per-turn charge that grows with context length even under caching, while the memory system’s per-turn read cost remains roughly fixed after a one-time write phase. At a context length of 100k tokens, the memory system becomes cheaper after approximately ten interaction turns, with the break-even point decreasing as context length grows. These results characterize the accuracy–cost trade-off between the two approaches and provide a concrete criterion for selecting between them in production deployments. Executive Summary When people use an AI assistant over many days or months — for customer support, personal help, or learning — the AI needs to remember what was said in earlier conversations. There are two main ways to do this. The first approach keeps a full record of every past conversation and sends the whole record to the AI every time a new question is asked. The second approach saves only the key facts from past conversations into a small memory store, and pulls out only what is needed when a question comes in. Both approaches work, but they have different strengths and costs. This paper compares the two to help organizations decide which one is right for their situation. We tested both approaches using three separate tests that measure how well an AI assistant remembers things about a user over time — such as facts, dates, and personal details. We also tracked how much each approach costs as users ask more and more questions, taking into account price discounts that AI service providers offer when the same conversation is sent repeatedly, which is called a prompt caching discount . On accuracy, the full-history approach was more correct on two of the three tests, by a margin of about 33 to 35 percentage points. On the third test — which checks whether the AI stays consistent about a user’s personal details, such as their habits and preferences — the memory-based approach performed just as well and was even slightly better in one comparison. This means that the full-history approach is not always the winner; it depends on what kind of questions are being asked. On cost, the picture changes as users interact more. In the first few interactions, the full-history approach costs less. However, because it sends the entire conversation history every time, its cost keeps growing with each new question. The memory-based approach pays a small one-time cost upfront to build the memory store, and then charges only a small, steady fee for each new question. When a conversation history reaches about 100,000 words, the memory-based approach becomes cheaper after around 10 questions, and saves about 26% of the total cost by the time 20 questions have been asked. The longer the conversation history grows, the sooner this saving comes into effect. In short, there is no single best approach for every situation. For services where users come back many times — such as ongoing customer support, personal assistants, or tutoring tools — the memory-based approach saves a meaningful amount of money while still giving good answers. For one-time or short interactions where accuracy is the top priority, the full-history approach is the better choice. This paper gives organizations a simple way to calculate which approach will cost less, based on how many times their users are likely to interact with the AI. 1 Introduction As conversational AI assistants are deployed in settings that span multiple sessions—personal assistants, customer-support agents, educational tutors—they must maintain coherent access to information from past interactions. Two strategies address this requirement. The first passes the raw conversation history directly into the context window of a long-context large language model (LLM), relying on the model to process all prior turns on every query. The second builds a dedicated memory system that distills interactions into compact, structured records and retrieves only the relevant facts at query time Chhikara et al. 2025 ; Maharana et al. 2024 . As context windows have expanded to millions of tokens Google DeepMind 2025 ; Singh et al. 2025 , some have questioned whether retrieval-based approaches remain necessary Khalusova 2024 ; Rengifo and Martin 2025 . However, the practical case for memory systems rests not only on accuracy but on the economics of long-running deployments. API pricing for long-context LLMs scales with input length, and re-submitting a 100k-token history on every user turn accumulates cost rapidly. Prompt caching partially alleviates this by discounting tokens that share a common prefix with a recently processed prompt Gim et al. 2024 —OpenAI charges cached input tokens at 10% of the standard rate for GPT-5-series models, and Anthropic applies the same 90% discount across its Claude models OpenAI 2024 ; OpenAI 2025a ; Anthropic 2024 —but each turn still incurs an incremental charge proportional to context length. Memory systems have a structurally different cost profile: a one-time write phase covers fact extraction and embedding, while each subsequent query consumes only the retrieved facts, which are far smaller than the full history. Whether the memory approach is economically preferable depends on how frequently a user re-engages with the same context—a quantity we refer to as the break-even point . In this paper, we compare a fact-based memory system built on the Mem0 framework Chhikara et al. 2025 against long-context LLM inference on three memory-centric benchmarks: LongMemEval, LoCoMo Maharana et al. 2024 , and PersonaMem v2. In addition to measuring factual recall accuracy, we construct a cost model that incorporates prompt caching and tracks cumulative API expenditure as a function of the number of interaction turns. Our main contributions are: • An accuracy comparison across three benchmarks of varying context length and question type, showing that long-context LLMs achieve higher factual recall on most datasets, while the memory system remains competitive on persona-consistency tasks. • A cost model incorporating prompt caching that demonstrates the memory system’s fixed per-turn read cost is structurally different from the cached but still growing per-turn cost of long-context inference. • A break-even analysis showing that at a context length of 100k tokens the memory system becomes cheaper after approximately ten interaction turns, and that this threshold decreases as context length grows. 2 Related Work Long-context LLMs. Extending the effective context window of transformer-based models has been studied through architectural changes Guo et al. 2022 ; Beltagy et al. 2020 and positional encoding techniques Chen et al. 2023b . Recent models such as Gemini 2.5 Pro Google DeepMind 2025 ,

Read the complete text