Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

Yakov Pyotr Shkolnikov · 2026

Read Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices by Yakov Pyotr Shkolnikov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04428.txt.

Description

Shkolnikov persists agent KV caches to disk for edge multi-agent systems. RAM cannot hold every cache at once: on Apple M4 Pro, 3 agents at 8K context fill FP16 memory. A 10 agent workflow thus evicts constantly, and each eviction forces a full re-prefill. His system stores per agent Q4 caches in safetensors format and reloads them straight into attention. Cross phase injection accumulates attention state without recomputation. Restoration cuts time to first token by up to 136 times, fits 4 times more contexts than FP16, and shifts perplexity only slightly.

Catalog record

CREATORYakov Pyotr Shkolnikov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73951
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04428

Browse the full library · About the Tower of Babel