The Missing Memory Hierarchy: Demand Paging for LLM Context Windows Conference: ACM SIGOPS 31st Symposium on Operating Systems Principles; 2017; Shanghai, China CCS: Software and its engineering Allocation / deallocation strategies CCS: Software and its engineering Virtual memory CCS: Computing methodologies Natural language processing

Tony Mason · 2026

Read The Missing Memory Hierarchy: Demand Paging for LLM Context Windows Conference: ACM SIGOPS 31st Symposium on Operating Systems Principles; 2017; Shanghai, China CCS: Software and its engineering Allocation / deallocation strategies CCS: Software and its engineering Virtual memory CCS: Computing methodologies Natural language processing by Tony Mason (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09023.txt.

Introduction

Tony Mason presents Pichay, which pages LLM context on demand. The context window acts as L1 cache with no L2 or virtual memory, so stale tool results occupy context for the session and structural waste runs near 22 percent. As a transparent proxy, Pichay evicts stale content, faults pages back on re-request, and pins working-set pages, cutting context use by up to 93 percent live.

Catalog record

CREATORTony Mason
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69004
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09023

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Missing Memory Hierarchy: Demand Paging for LLM Context Windows Conference: ACM SIGOPS 31st Symposium on Operating Systems Principles; 2017; Shanghai, China CCS: Software and its engineering Allocation / deallocation strategies CCS: Software and its engineering Virtual memory CCS: Computing methodologies Natural language processing Tony Mason Abstract . The context window of a large language model is not memory. It is L1 cache—a small, fast, expensive resource that the field treats as the entire memory system. There is no L2, no virtual memory, no paging. Every tool definition, every system prompt, and every stale tool result occupies context for the lifetime of the session. The result is measurable: across 857 production sessions and 4.45 billion effective input tokens, 21.8% is structural waste. We present Pichay , a demand paging system for LLM context windows. Implemented as a transparent proxy between client and inference API, Pichay interposes on the message stream to evict stale content, detect page faults when the model re-requests evicted material, and pin working-set pages identified by fault history. In offline replay across 1.4 million simulated evictions, the fault rate is 0.0254%. In live production deployment over 681 turns, the system reduces context consumption by up to 93% (5,038KB → \to 339KB); under extreme sustained pressure, the system remains operational but exhibits the expected thrashing pathology, with repeated fault-in of evicted content. The key observation is that the problems the field faces—context limits, attention degradation, cost scaling, lost state across sessions—are virtual memory problems wearing different clothes. The solutions exist: working set theory ( 3 ) , demand paging, fault-driven replacement policies, and memory hierarchies with multiple eviction-managed levels. We describe the architecture of a full memory hierarchy for LLM systems (L1 through persistent storage), report on the first three levels deployed in production use (L1 eviction, L2 fault-driven pinning, L3 model-initiated conversation compaction), and identify cross-session memory as the remaining frontier. 1. Introduction In 1961, the Atlas computer introduced demand paging: programs could address more memory than physically existed, and the hardware would transparently load pages from backing store as needed ( 8 ) . Before Atlas, programmers manually managed overlays—explicitly swapping code segments in and out of a fixed address space. The transition from overlays to virtual memory is one of the foundational advances in systems architecture. Large language model context windows are in the overlay era. Every agentic AI tool—Claude Code, Cursor, GitHub Copilot, Windsurf—manually assembles context on every API call: tool definitions, system prompts, and the complete message history. There is no eviction policy, no demand loading, no working set estimation. The default behavior is accumulation until the context limit forces a crisis. This paper makes two contributions. First, we provide the empirical evidence that context management is a memory management problem. We instrument 857 production sessions (54,170 API calls, 4.45 billion effective input tokens) and find that 21.8% of tokens are structural waste from three sources: unused tool schemas (11.0%), duplicated content (2.2%), and stale tool results (8.7%) reprocessed at a median amplification of 84.4 \times . These are not implementation bugs. They are the inevitable consequence of managing a finite resource (the context window) without the abstractions that operating systems developed for the analogous problem fifty years ago. Second, we build the system. Pichay is a demand paging system for LLM context windows, implemented as a transparent proxy between client and inference API. It distinguishes garbage collection (removing ephemeral tool output that cannot be re-requested) from paging (evicting addressable content that can be faulted back in). It detects page faults when the model re-requests evicted content. It pins working-set pages identified by fault history. And the eviction summaries it generates—designed as space-saving markers—function as retrieval handles that models understand without instruction: “ [Paged out: Read file.py (8,192 bytes). Re-read if needed.] ” In production deployment, the system extends a session from 7% free context to 43% free—the difference between imminent context death and substantial remaining capacity. Over a 681-turn session, it sustains operation at 97% eviction rate, though this extreme case reveals the thrashing pathology well-known from OS research ( 3 ) : when the working set exceeds the resident set, the system spends more effort on faulting than on useful work. The deeper observation is that the context window is not memory. It is L1 cache. And nobody is building the rest of the hierarchy. The field’s response to context limits is to make L1 bigger—1M token windows, 10M token windows. This is the equivalent of building machines with more physical RAM instead of inventing virtual memory. It works, but it doesn’t scale, and it misses the architectural insight: a managed hierarchy of small-fast-expensive and large-slow-cheap storage, connected by eviction policies and fault mechanisms, outperforms any single level no matter how large. We describe the full hierarchy: L1 (the generation window), L2 (the working set, demand-paged and pinned), L3 (session history, compressed with declared losses), and L4 (cross-session persistent memory, indexed and retrievable). The first two levels are deployed and evaluated. L3 is implemented—the model can compress conversation turns into outcome summaries via cooperative cleanup tags—but not yet evaluated at scale. L4 is designed and its mechanisms identified. The contributions: (1) A large-scale empirical characterization of context window utilization in a production agentic AI corpus, establishing that 21.8% of tokens are structural waste with a measured taxonomy. (2) Pichay , a demand paging system for LLM context windows with empirically measured fault rate (0.0254% offline, deployed in production). (3) Fault-driven pinning: a page replacement policy that learns from its own mistakes—one fault pins a page for the session, reducing repeated faults in steady-state workloads. (4) Cooperative memory management via two side channels (phantom tools and cleanup tags): mechanisms enabling the model to voluntarily release cold pages, request cached faults, and compress conversation history—a new point in the design space not available in hardware memory hierarchies, where applications are non-cooperative. (5) The architectural observation that LLM context management maps structurally (not merely metaphorically) to virtual memory, and that the full OS memory hierarchy—from cache replacement policies to working set theory—applies directly. (6) Design of a four-level memory hierarchy for LLM systems, with the first three levels deployed (L1 eviction, L2 pinning, L3 cooperative compaction) and the first two evaluated. 2. Background 2.1. Agentic AI Architecture Modern AI coding assistants operate as agentic systems: the model receives a set of tool definitions (JSON schemas describing available operations), a system prompt (identity, instructions, behavioral constraints), and a growing message history (alternating user and assistant turns including tool invocations and their results). On each API call, the client application assembles the full context: system prompt, tool definitions, and complete message history. The inference provider tokenizes this context, computes attention over all tokens, and generates a response. The response is appended to the message history, and the cycle repeats. The critical property: context is reassembled from scratch on every API call. There is no persistent state at the inference layer between calls. The entire conversation—including tool definitions that haven’t changed and results that will never be referenced again—is sent, tokenized, and attended to on every turn. 2.2. Context Windows as Physical Memory The analogy between context windows and physical memory is structural, not merely metaphorical (Table

Read the complete text