Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
Yakov Pyotr Shkolnikov · 2026
Read Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices by Yakov Pyotr Shkolnikov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04428.txt.
Description
Shkolnikov persists agent KV caches to disk for edge multi-agent systems. RAM cannot hold every cache at once: on Apple M4 Pro, 3 agents at 8K context fill FP16 memory. A 10 agent workflow thus evicts constantly, and each eviction forces a full re-prefill. His system stores per agent Q4 caches in safetensors format and reloads them straight into attention. Cross phase injection accumulates attention state without recomputation. Restoration cuts time to first token by up to 136 times, fits 4 times more contexts than FP16, and shifts perplexity only slightly.
Catalog record
| CREATOR | Yakov Pyotr Shkolnikov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73951 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04428