Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs
Andy Zeyi Liu, Michael Zhang, Ilana Greenberg, Adam Alnasser, Lucas Baker, John Sous · 2026
Read Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs by Andy Zeyi Liu, Michael Zhang, Ilana Greenberg, Adam Alnasser, Lucas Baker, John Sous (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06225.txt.
Introduction
Andy Zeyi Liu and five co-authors steer language models by editing the latent attention cache instead of the visible prompt. Prompting gives strong control but stores guidance tokens at every layer and clutters long conversations. Activation steering is compact, yet usually weaker and poor at carrying large structured reminders over time.
Memory Inception inserts text derived key value banks at selected layers only, treating steering as selective cache allocation. On matched personality steering tasks it gave the best balance of control against drift, rivaling prompting and beating a leading activation method. It also supports mid conversation behavior shifts without rewriting the transcript. On structured reasoning tests it beat visible prompting in most settings while cutting matched cache storage by up to 118 times.
Catalog record
| CREATOR | Andy Zeyi Liu, Michael Zhang, Ilana Greenberg, Adam Alnasser, Lucas Baker, John Sous |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06225
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs Andy Zeyi Liu, Michael Zhang, Ilana Greenberg, Adam Alnasser, Lucas Baker, John Sous Abstract Steering large language models (LLMs) is usually done by either instruction prompting or activation steering. Prompting often gives strong control, but caches guidance tokens at every layer and can clutter long interactions; activation steering is compact but typically weaker and does not support large structured reminders. We introduce memory inception (MI), a training-free method that steers in latent attention space by inserting text-derived key-value (KV) banks only at selected layers. Rather than materializing reminder content throughout the prompt cache, MI treats steering as selective KV allocation, injecting latent slots only where the model routes to them. On matched personality-steering tasks, MI gives the best overall control–drift trade-off, remaining competitive with prompting while consistently outperforming CAA. On updateable guidance, MI supports mid-conversation behavior shifts without rewriting the visible transcript, achieving the highest post-shift alignment on Qwen3. On structured reasoning, MI outperforms visible prompting on HARDMath and PHYSICS (10/12 subject × \times mode cells), serving as proxies for structured reasoning in verifiable domains, while cutting content-matched KV storage by up to 118 × \times . These results position MI as a powerful steering method when guidance is persistent, structured, or expensive to keep in the visible transcript. 1 Introduction Language models often need persistent guidance rather than one-shot instructions. A user may want an assistant to maintain a personality trait across a questionnaire, preserve a tone across a multi-turn dialogue, keep a long-context fact active while answering later questions, or apply a compact checklist of reasoning heuristics to a domain benchmark. The standard solution is prompting, where reminder tokens are inserted into visible context and cached at every layer. A second family, activation steering, changes hidden states directly through learned or contrastive residual directions. These two interfaces make opposite trade-offs: prompting exposes the reminder text and stores it everywhere, while activation steering is compact but not naturally text-faithful or easily updateable. We study a third interface: latent attention state . We introduce memory inception (MI), illustrated in Figure 1 , a training-free method that encodes a descriptor, summary, retrieved fact, or task heuristic into a bank of latent key-value slots, and lets only selected attention layers/heads attend over those slots alongside the ordinary prompt tokens. The reminder is therefore available where attention reads it, rather than being repeatedly materialized as visible text or injected as a query-independent residual offset. Prompt steering stores guidance-derived KV states at every layer, even when only a small subset of layers drives control; MI instead treats guidance as a selective KV-allocation problem: identify the layers and attention units where the reminder matters, then insert latent slots only there. The layer-wise KV footprint shrinks from all L layers to L_{\mathrm{ctrl}} controlled layers. Figure 1: Memory inception (MI). Concise and compact reminder text is encoded into latent KV memory-bank slots and appended only at automatically chosen layers, heads, or KV groups during decoding. Unlike prompt steering, the reminder stays outside the visible transcript at inference time and need not be stored at every layer; unlike CAA-style activation steering, the intervention acts through query-dependent attention over latent reminder slots rather than a global residual-vector offset. Conceptually, prompting and activation steering sit at opposite ends of two design axes: visible vs. latent control content, and global vs. selective intervention. MI is in the middle ground, text-conditioned but hidden from the visible transcript, and localized to the attention sites that route to it. For implicit signals such as personality, tone, or reasoning heuristics, the question becomes not whether the reminder can be cached everywhere, but where it actually needs to live. We evaluate MI in three regimes. First, on matched personality-steering tasks where prompting, CAA, and MI receive the same control signal, we test whether latent KV steering improves the control–drift trade-off. Second, on mid-dialogue behavior shift, we test whether guidance can stay active or be updated without rewriting the visible transcript. Third, on HARDMath ( 4 ) and PHYSICS ( 5 ) , we test whether the same mechanism carries reusable reasoning-heuristic banks rather than short style descriptors. Contributions: • Steering interface. We introduce memory inception or MI, a training-free method that injects text-conditioned latent KV banks only at selected layers rather than materializing reminder text throughout the full prompt cache. • Interpretation and implementation. We formulate the method as bank-level attention routing over prompt, target, reference, and auxiliary memories, and we give a canonical-key RoPE implementation that supports dense attention, grouped-query attention, and Qwen3-style MoE models. • Empirical positioning. On personality-steering tasks, MI remains competitive with prompting while consistently outperforming CAA on control–drift trade-offs. In reasoning domains, MI acts as a training-free alternative to fine-tuning, improving PHYSICS on average and consistently outperforming the plain model on HARDMath. • Operating systems value. Memory banks are most useful when guidance is persistent or structured, enabling reduction in KV storage by a factor of 6–118 \times . 2 Related Work We keep the main-text discussion focused on the methods most directly comparable to MI and in Appendix A we list additional related works. Prompting and latent prompts. Prompting remains the default inference-time interface for steering language models, from in-context instructions to chain-of-thought scaffolds ( 3 ; 26 ; 10 ) . A closely related family of methods learns continuous prefixes or soft prompts that are injected into the model without changing base weights ( 16 ; 13 ) . MI shares the same goal of inference-time control, but differs in two ways: it is training-free, and it constructs the steering state directly from user-provided text or task heuristics at inference time rather than from optimized prompt parameters. Activation, attention, and cache steering. Activation Addition and CAA steer models by adding residual directions derived from prompt pairs or positive/negative examples ( 23 ; 19 ) . Representation-engineering and function-vector work likewise shows that reusable latent directions can capture high-level behaviors or in-context functions ( 31 ; 22 ) . More recent attention-level methods, such as AutoPASTA and Spotlight, steer by reallocating attention toward instruction tokens ( 30 ; 25 ) , and KV Cache Steering manipulates cached prompt representations to induce reasoning behavior ( 1 ) . MI differs from these lines by appending new text-derived KV content at selected sites, so the model can attend to reusable blocks of guidance instead of compressing them into a single residual direction or a bias over visible prompt tokens. 3 Memory Inception: Formalism & Implementation Overview. Memory inception (MI) is a training-free steering interface that encodes reminder content into latent KV banks and attaches those banks only at a small set of selected layers and attention sites, where the model can attend to them alongside ordinary prompt tokens. Figure 1 illustrates the pipeline; the following subsections formalize the intervention, bank construction, and KV-cache budget in turn. 3.1 Memory Inception in Latent Attention Space We begin with the intervention interface itself: what a selected attention site sees, where the additional banks are attached, and how routing changes when more than one memory bank is present. Attention. Consider one attention site in a decoder-only transformer ( 24 ) . At generation step t , let