CoMem : Context Management with A Decoupled Long-Context Model

Yuwei Zhang, Chengyu Dong, Shuowei Jin, Changlong Yu, Hejie Cui, Hongye Jin, Xinyang Zhang, Hamed Bonab, Colin Lockard, Jianshu Chen, Zhenyu Shi, Jingbo Shang, Xian Li, Bing Yin · 2026

Read CoMem : Context Management with A Decoupled Long-Context Model by Yuwei Zhang, Chengyu Dong, Shuowei Jin, Changlong Yu, Hejie Cui, Hongye Jin, Xinyang Zhang, Hamed Bonab, Colin Lockard, Jianshu Chen, Zhenyu Shi, Jingbo Shang, Xian Li, Bing Yin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30842.txt.

Introduction

Zhang and colleagues present CoMem, splitting memory management from the agent workflow so the two can run together. Context handling lets agentic models solve long-horizon tasks through past recaps. But this incurs large reading overhead for extra summary tokens. CoMem uses a k-step-off pipeline overlapping memory recaps with agent inference, effectively masking context-processing latency. A reward-driven training strategy aligns the memory model to capture statistics for agent decisions. On SWE-Bench-Verified, CoMem provides 1.4 times latency improvements while preserving performance.

Catalog record

CREATORYuwei Zhang, Chengyu Dong, Shuowei Jin, Changlong Yu, Hejie Cui, Hongye Jin, Xinyang Zhang, Hamed Bonab, Colin Lockard, Jianshu Chen, Zhenyu Shi, Jingbo Shang, Xian Li, Bing Yin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53114
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30842

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoMem : Context Management with A Decoupled Long-Context Model Yuwei Zhang, Chengyu Dong, Shuowei Jin, Changlong Yu, Hejie Cui, Hongye Jin, Xinyang Zhang, Hamed Bonab, Colin Lockard, Jianshu Chen, Zhenyu Shi, Jingbo Shang, Xian Li, Bing Yin Abstract Context management enables agentic models to solve long-horizon tasks through iterative summarization of previous interaction histories. However, this process typically incurs substantial decoding overhead for the extra summarization tokens, which significantly affect the end-to-end response latency at deployment. In this paper, we introduce CoMem , a novel framework that decouples memory management from the primary agent workflow, enabling these processes to execute in parallel. We propose a k k -step-off asynchronous pipeline that overlaps the memory model’s summarization with the agent’s inference, effectively masking the latency of context processing. To ensure robustness under this asynchronous setting, we introduce a reward-driven training strategy that aligns the memory model to capture sufficient statistics for the agent’s decision-making. Theoretical analysis confirms that CoMem offers a superior efficiency-effectiveness trade-off compared to coupled architectures. Our extensive experimental results on SWE-Bench-Verified show that CoMem provides 1.4x latency improvements upon vanilla long-context solutions while preserving most of the performance. Furthermore, we demonstrate that these latency gains scale favorably with increased system throughput, offering a modular path forward for the independent optimization of agent reasoning and memory compression. † † affiliationnotice: Equal contribution 1 Introduction The capabilities of Large Language Model (LLM) agents have expanded significantly, moving beyond simple conversational assistants to autonomous systems capable of tackling complex, multi-step problems in domains such as software engineering ( 8 ) , scientific discovery ( 13 ) , and open-ended exploration ( 5 ; 21 ) . Unlike standard question-answering tasks, the success on these applications depends not only on the immediate instruction but on the agent’s ability to maintain a coherent understanding of its entire interaction history. For instance, in repository-level code generation, an agent must recall decisions made thousands of steps prior to generate valid subsequent actions. Consequently, the ability to process and reason over extremely long contexts has become a non-negotiable requirement for high-performing agentic systems ( 33 ; 25 ; 14 ; 17 ; 29 ) . Despite the increasing attention to extend the effective context window, there has been surprisingly little attention paid to the severe computational costs introduced by long-context processing during agentic inference. While modern LLMs can natively accept context windows spanning millions of tokens, utilizing them during online agent deployment is often prohibitively expensive ( 30 ) . The primary challenge lies in the inference latency of the decoding stage. As the interaction history grows, the KV cache footprint expands linearly, and the attention computation cost scales, saturating the High Bandwidth Memory (HBM) of modern GPUs. Unlike the prefilling phase, which is compute-bound and parallelizable, the auto-regressive decoding phase is memory-bound, since fetching the entire KV cache for every generated token results in high latency that degrades the user experience and limits the throughput of real-time systems ( 19 ) . To address these computational challenges, previous solutions have largely pursued two directions: context reduction and system-level optimization. Context reduction strategies attempt to limit the number of tokens the model must attend to. Sliding-window attention ( 4 ) restricts the agent to a fixed local context, inherently sacrificing the ability to recall distant dependencies, such as the initial user goal or a bug identified thousands of steps prior. Retrieval-Augmented Generation (RAG) ( 24 ; 16 ) alleviates this by fetching relevant snippets from the history. However, RAG relies heavily on semantic similarity, which often fails to capture the high-level state or procedural trajectory of an agent, leading to fragmented reasoning. On the other hand, system-level optimizations aim to make processing the full context more efficient without reducing the number of tokens. Modern inference engines like vLLM ( 11 ) have revolutionized memory management through PagedAttention, significantly reducing fragmentation. Similarly, Sparse Attention mechanisms ( 27 ) and KV Cache Quantization ( 6 ) reduce the compute and memory bandwidth requirements by sparsifying interactions or lowering numerical precision. While these methods provide substantial throughput improvements, they do not solve the the redundancy of re-encoding or re-attending to a massive, largely static interaction history for every single decoding step. As a result, even highly optimized engines eventually hit a latency wall as the interaction trace grows indefinitely. To bridge this gap, we introduce CoMem , a framework that decouples the distinct responsibilities of memory management and agentic reasoning. Unlike prior approaches that force a single large model to handle both history compression and policy generation, CoMem offloads the heavy lifting of long-context processing to a dedicated, lightweight summarization model. This architectural separation allows us to propose a novel k -step-off asynchronous pipeline, where the memory model continuously compresses history in the background, freeing the main agent to decode with a significantly reduced context window. To ensure this compressed state effectively guides the agent, we introduce a reward-driven alignment strategy that trains the memory model to capture the ”sufficient statistics” required for optimal decision-making. By shifting the computational burden from the critical path of decoding to a parallelizable background process, CoMem achieves the best of both worlds: the global reasoning capability of long-context models and the low-latency inference of short-context systems. Our contributions are summarized as follows: • We propose CoMem , a framework that separates memory management from reasoning, enabling the use of specialized, lightweight models for efficient history compression. • Asynchronous Inference: We design a k -step-off pipeline that overlaps memory summarization with agent execution, masking the latency of context processing and reducing the decoding overhead by up to 1.4\times . • We introduce a novel training methodology that aligns the memory model using a functional equivalence reward, ensuring the compressed summary recovers the full-context agent’s policy without expensive online interaction. • Extensive experiments on SWE-Bench-Verified demonstrate that CoMem significantly reduces latency while maintaining competitive performance against state-of-the-art long-context baselines. Code is available at: https://github.com/horizon-llm/CoMem.git . Figure 1 : CoMem framework: a decoupled agent framework that offloads long-context compression to an asynchronous, lightweight memory model, significantly reducing inference latency without compromising reasoning performance. 2 Preliminaries 2.1 Standard Agent Standard agent interaction is a Partially Observable Markov Decision Process (POMDP), where each process starts from an initial observation o_{1} that usually contains a problem statement, and then a multi-turn interaction history of length T is generated as \tau=(o_{1},a_{1},o_{2},a_{2},\ldots,o_{T}) where a_{i} usually contains a reasoning block and a tool call, and o_{i} usually represents an observation returned from the environment. The standard Large Language Model (LLM) agent takes the entire interaction history \tau as input at each step to infer the internal belief state of the environment, and generates the next action a_{i} . This results in a linearly scaling context length during the interaction that pressures the model’s inference efficiency and performance. 2.2 LLM Inference Modern LLMs usually employ a two-stage inference process: Prefill Stage and Decode Stage . The Prefill Stage serves as the initial step where the key-value cache (KV cache) for the prompt (also called prefix) sequences are built for each KV head in the Transformer architecture 1 1 1 Note that here we discriminate KV head from attention head because of Grouped Query

Read the complete text