Cooperative Memory Paging with Keyword Bookmarks for Long-Horizon LLM Conversations

Ziyang Liu · 2026

Read Cooperative Memory Paging with Keyword Bookmarks for Long-Horizon LLM Conversations by Ziyang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12376.txt.

Introduction

Ziyang Liu proposes cooperative paging for long chats that outgrow a language model's working memory. Evicted passages are replaced by short keyword bookmarks. The model calls a recall tool to fetch a full passage when needed. On ten long multi-session conversations, the method beat truncation, search, and full context. Answer quality led on all four models tested. Picking the right bookmark remained the weak point, however.

Catalog record

CREATORZiyang Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54542
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cooperative Memory Paging with Keyword Bookmarks for Long-Horizon LLM Conversations Ziyang Liu Abstract When LLM conversations grow beyond the context window, old content must be evicted—but how does the model recover evicted information when it needs it? Existing approaches either require the model to self-diagnose information gaps via explicit retrieval calls, detect faults only for tool outputs like file reads, or compress content irreversibly. We propose cooperative paging : evicted conversation segments are replaced with minimal keyword bookmarks ( [pN:keywords] , ∼ \sim 8 tokens each), and the model is given a recall() tool to retrieve full content on demand. The bookmarks serve as a lightweight table of contents for compressed memory, enabling the model to decide when retrieval is needed. On the LoCoMo benchmark (10 real multi-session conversations, 300+ turns each), cooperative paging achieves the highest answer quality among six methods tested—outperforming truncation, BM25 retrieval, word-overlap retrieval, a search-tool baseline, and truncated full context—on four models from three provider families (GPT-4o-mini, DeepSeek-v3.2, Claude Haiku, GLM-5), a ranking confirmed by four independent LLM judges ( p = 0.017 p{=}0.017 vs. BM25, paired bootstrap). We then study the paging design space directly with a turn-by-turn simulator (3,176 probes on synthetic long conversations, 1,600 on LoCoMo): a 5 × \times 4 ablation over page-boundary strategies (fixed-size, topic-shift, exchange-count) and eviction policies (FIFO, LRU, LFU, Bélády oracle). We find that (1) page granularity dominates : coarse fixed-size pages ( fixed_20 ) reach 96.7% while content-aware topic_shift over-fragments and collapses to 56.7%; (2) eviction policies are data-dependent : Bélády upper-bounds online policies by 8–14 points, and which online policy wins flips between forward-moving synthetic data (FIFO) and revisit-heavy LoCoMo (LFU); (3) the bookmark generation bottleneck is partially closable : of six bookmark generation strategies we test, two improve over the heuristic baseline—hybrid (heuristic + LLM-discriminative augmentation) on synthetic data ( + 4.4 +4.4 E2E points) and llm-batch (single-call cross-page non-overlap) on real LoCoMo dialogues ( + 8.7 +8.7 points, + 50 +50 on open-domain questions). The other four strategies, including TF ⋅ \cdot IDF and per-page LLM, under-perform the heuristic by lowering the recall trigger rate—a structural failure mode predicted by the information-gap hypothesis; (4) the bookmark is now the bottleneck : the model triggers recall() 96% of the time but selects the correct page only 57% of the time when bookmarks are insufficiently distinctive. A format ablation on controlled probes ( n = 22 n{=}22 ) suggests that minimal keyword bookmarks achieve the best accuracy-to-cost ratio, and keyword specificity alone accounts for a 25 percentage point accuracy difference. 1 Introduction Large language models deployed in multi-turn conversations face a fundamental resource management problem: the context window is finite, but conversation history grows without bound. When the context budget is exceeded, old content must be evicted. The question is: how does the model get evicted content back when it needs it? This problem is the memory management problem that operating systems solved decades ago with virtual memory and demand paging ( 4 ) . In OS paging, the application is unaware of physical memory limits; the OS transparently loads pages from disk when needed. Recent work has applied this analogy to LLMs: 13 introduced hierarchical memory tiers with explicit retrieval via function calls, and 11 built a transparent paging proxy for agentic coding sessions. However, existing approaches have fundamental limitations. MemGPT requires the model to self-diagnose information gaps—but models cannot reliably detect what they do not know. Pichay detects faults only through tool-call matching, which is limited to addressable outputs like file reads. Compression approaches like Adaptive Focus Memory ( 3 ) lose details irreversibly. We propose cooperative paging : when conversation content is evicted, it is replaced with a minimal keyword bookmark costing only {\sim} 8 tokens, e.g.: [p3: allergy, peanut, budget] The model is given a recall() tool to retrieve the full content of any bookmarked segment on demand. Unlike MemGPT, where the model must formulate free-form search queries about what it might be missing, our bookmarks provide a table of contents for compressed memory: the model sees which topics are available and decides when to retrieve. This shifts the cognitive load from “guess what you might need” to “check the index.” We evaluate cooperative paging on both controlled experiments and the LoCoMo benchmark ( 10 ) —a dataset of real multi-session conversations spanning 300+ turns. Our contributions are: 1. A cooperative paging mechanism using minimal keyword bookmarks and a recall tool. On controlled scenarios, the model achieves 91% recall accuracy with zero observed false positives ( n=22 ). On LoCoMo, Bookmark+Recall achieves the highest answer quality among six methods on both GPT-4o-mini and DeepSeek-v3.2 (§ 4 ). 2. A paging design study that treats page boundary and eviction policy as first-class design choices. Using a turn-by-turn paging simulator, we run a full 5 \times 4 ablation (3,176 synthetic probes, 1,600 LoCoMo probes) and find that page granularity dominates (fixed-size 96.7% vs. topic-shift 56.7%), Bélády’s oracle beats FIFO by 14 points, and the remaining bottleneck has shifted from “when to recall” (now 96% correct) to “which page to recall” (57% correct). To our knowledge, this is the first study of page boundary detection and eviction policies for in-session LLM conversation paging (§ 5 ). 3. Design principles for bookmark construction: minimal keyword bookmarks ( \sim 8–24 tokens) achieve the best accuracy-to-cost ratio over richer formats, and keyword specificity is the most impactful design factor in our controlled study, accounting for a 25.7 percentage point improvement (§ 6 ). 4. Closing the bookmark bottleneck. We propose two complementary bookmark generation strategies— hybrid (heuristic + LLM-discriminative augmentation) and llm-batch (single-call cross-page non-overlap)—that improve end-to-end accuracy by +4.4 points on synthetic data and +8.7 points on real LoCoMo dialogues, with a +50 point improvement on open-domain questions. Critically, four other natural strategies (TF \cdot IDF, per-page LLM, etc.) under-perform the heuristic baseline by lowering the recall trigger rate—a structural failure mode we link back to the information-gap hypothesis (§ 7 ). 2 Why Passive Fault Detection Fails 2.1 Hypothesis A natural approach to detecting memory faults is to monitor the model’s output uncertainty. When generating a response, if the model needs information that was evicted, its token-level NLL should spike—analogous to a hardware page fault in OS virtual memory. EM-LLM ( 5 ) demonstrated that Bayesian surprise (a form of prediction error) effectively segments input text into episodic events. We hypothesized that the same signal, applied during output generation, could detect when the model lacks evicted context. 2.2 Experiment We constructed 5 multi-turn conversations, each containing planted critical information (food allergies, budget constraints, scheduling conflicts). For each conversation, we generated responses under two conditions: (A) full context with all information present, and (B) gist-compressed context where the critical information was replaced with a short summary. We measured per-token NLL using logprobs from the model API and compared rolling NLL windows ( w=32 tokens) between conditions. 2.3 Results Figure 1: NLL-based fault detection fails. (a) Mean NLL under full vs. gist context across 5 test cases. (b) \Delta NLL (gist - full): two cases show reverse signal (gist is more confident), and the mean experimental \Delta NLL (+0.012) is indistinguishable from the control (+0.025). Table 1

Read the complete text