Application-Layer Dual Memory for Conversational AI: Achieving Virtually Unbounded Context Without Model Modification
Rajendra Narayan Jena Rajan Padmanabhan Sankar Arumugam Infosys Limited Infosys Limited Infosys Limited [email protected] [email protected] [email protected] · 2026
Read Application-Layer Dual Memory for Conversational AI: Achieving Virtually Unbounded Context Without Model Modification by Rajendra Narayan Jena Rajan Padmanabhan Sankar Arumugam Infosys Limited Infosys Limited Infosys Limited [email protected] [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20724.txt.
Introduction
Rajendra Narayan Jena, Rajan Padmanabhan, and Sankar Arumugam present CALMem, an application-layer dual memory for conversational AI. Language models work within fixed context windows, so long talks lose history when compaction discards it and sessions reset memory to zero. CALMem needs no model change and pairs an episodic layer of sliding-window vector embeddings with a semantic layer of agent-written structured facts. Together they give assistants virtually unbounded effective context.
Catalog record
| CREATOR | Rajendra Narayan Jena Rajan Padmanabhan Sankar Arumugam Infosys Limited Infosys Limited Infosys Limited [email protected] [email protected] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41031 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Application-Layer Dual Memory for Conversational AI: Achieving Virtually Unbounded Context Without Model Modification Rajendra Narayan Jena Rajan Padmanabhan Sankar Arumugam Infosys Limited Infosys Limited Infosys Limited [email protected] [email protected] [email protected] Abstract Large language models (LLMs) operate within fixed context windows that fundamentally limit conversational continuity. When context fills, compaction discards history irreversibly; when sessions end, all memory resets to zero. Existing solutions—larger context windows, retrieval-augmented generation for knowledge bases, and memory-augmented architectures such as MemGPT—either require model modification, impose provider lock-in, or do not address the compaction continuity problem. We present CALMem (Conversational Application-Layer Memory) , an application-layer dual memory architecture that gives LLM-based conversational assistants virtually unbounded effective context without any modification to the underlying model. CALMem combines two complementary memory subsystems: an episodic memory layer built on sliding-window vector embeddings of conversation history, and a semantic memory layer of agent-writable structured facts. A token-budget-adaptive injection mechanism, called the MOIM (Message of Injected Memory) , automatically retrieves and injects relevant past context each turn, scaling injection depth inversely with context pressure. A key contribution is intra-session retrieval : compacted-away turns from the current session remain searchable, closing a gap unaddressed by prior work. The system is implemented as a pure application layer in a production Rust codebase, is provider-agnostic, and degrades to original LLM behaviour with zero overhead when disabled. We describe the architecture, design decisions, and performance characteristics, and analyse the trade-offs that guided each implementation choice. 1 Introduction The context window is the fundamental unit of memory for a large language model. Everything the model “knows” during a conversation must fit within it. Current frontier models offer windows that have grown dramatically: GPT-5.2 provides 400,000 tokens OpenAI 2026 , Claude 4.6 Sonnet extends to 1,000,000 Anthropic 2026 , Gemini 2.0 Pro reaches 2,000,000 Google DeepMind 2026 , and Alibaba’s Qwen3-30B supports up to 1,000,000 tokens Alibaba Cloud 2026 . Despite these remarkable capacities, two structural problems remain unsolved. First, windows are still finite: long-running, task-intensive conversations—particularly in software engineering, research, or multi-day projects—exhaust even million-token windows. Second, and more fundamentally, every new conversation begins at zero regardless of window size. A 2,000,000-token window provides no memory of yesterday’s session. Practical deployments handle this through compaction : when the context approaches capacity, older turns are summarised and discarded from the active window. Compaction preserves the overall thread of conversation but loses detail. Once a turn is compacted away, any specific fact, decision, or piece of work it contained becomes inaccessible to the model—permanently, unless re-introduced through user effort. The cross-session problem is worse. Every new conversation begins from a blank slate. A user who spent three sessions last week debugging a complex system must re-explain the entire context at the start of session four. The model has no memory of any of it. These are not problems with model capability. They are structural consequences of the context window as an architectural primitive. The question is not whether to build larger windows—hardware and attention complexity impose real limits Vaswani et al. 2017 —but whether the application layer can compensate intelligently. There is also a third, practical problem: cost . LLM providers bill per token in context. Sending a 2,000,000-token window to the model on every turn costs proportionally more than sending a focused, compact context. For an application that manages memory intelligently—retrieving only the \sim 750 tokens of relevant history each turn rather than retaining millions of tokens in the active window—the per-turn API cost can be one to two orders of magnitude lower than the naïve “use the largest window available” approach. CALMem is not only a memory architecture; it is inherently a cost-efficiency architecture. Retrieval-Augmented Generation (RAG) Lewis et al. 2020 addresses this for static knowledge bases: pre-index a corpus, retrieve relevant passages at query time, inject into the prompt. But applying RAG to conversation history introduces problems not present in document retrieval. Conversation is not a corpus: it is a temporally ordered, contextually entangled stream. MemGPT Packer et al. 2023 introduced the idea of OS-inspired virtual context for LLMs, treating the context window as RAM and external storage as disk. This is a powerful framing, but it requires custom prompting infrastructure, relies on the model itself to make memory management decisions, and does not distinguish episodic from semantic memory. We propose a different approach: build the memory system entirely in the application layer, invisible to the model, requiring no prompt changes, no fine-tuning, and no modification to the underlying LLM provider. This paper makes the following contributions: 1. CALMem , a dual memory architecture combining episodic retrieval (vector embeddings of conversation history) and semantic retrieval (structured agent-writable facts), deployed as a pure application layer. 2. Token-budget-adaptive injection : a MOIM mechanism that scales retrieval depth proportionally to available context headroom, preventing memory injection from accelerating the compaction it is designed to mitigate. 3. Intra-session continuity : episodic retrieval operates over compacted-away turns from the current session, not just prior sessions— recovering context that no existing system retrieves. 4. Zero-regression design : the system degrades to standard LLM behaviour when disabled, with no latency overhead, no schema dependency, and no prompt structure changes. 5. A concrete implementation in a production Rust system with analysis of design decisions, performance characteristics, and observed trade-offs. The rest of the paper is organised as follows. Section 2 reviews related work. Section 3 describes the overall architecture. Sections 4 and 5 detail the two memory subsystems. Section 6 covers token-budget-adaptive injection. Section 7 discusses performance optimisation. Section 8 describes the implementation. Section 9 presents evaluation. Section 10 discusses limitations and future work. Section 11 concludes. 2 Related Work 2.1 Retrieval-Augmented Generation The original RAG framework Lewis et al. 2020 augments autoregressive language models with a non-parametric retrieval component: a dense passage retrieval (DPR) Karpukhin et al. 2020 model over a fixed document corpus. RAG was designed for open-domain question answering over static knowledge bases, not for conversational memory. Subsequent work has extended RAG to dialogue settings Xu et al. 2022 ; Zhong et al. 2023 , but these approaches typically treat conversation history as a flat document collection, losing the sequential and session structure that makes conversational memory different from document retrieval. 2.2 Long-Context Language Models A parallel line of work extends the context window itself through architectural innovations: sparse attention Child et al. 2019 , linear attention Katharopoulos et al. 2020 , and sliding window attention Beltagy et al. 2020 . More recently, FlashAttention Dao et al. 2022 makes very large windows computationally tractable. Despite these advances, the context window remains finite. More importantly, larger windows do not address the cross-session problem: each new conversation still begins with zero memory regardless of window size. 2.3 Memory-Augmented Language Models MemGPT Packer et al. 2023 introduced a compelling operating-system analogy: the context window as RAM, with a hierarchical external memory as disk, managed by the model itself through explicit function calls. This framework has inspired a family of agent memory architectures Sumers et al. 2023 ; Wang et al. 2023 . The key limitation is that memory management is delegated to the model. Generative Agents Park et al. 2023 demonstrated long-horizon memory for simulated social agents by combining reflection, synthesis, and retrieval. The system