Learned Structure in Cartridges : Keys as Shareable Routers in Self-Studied Representations
Maurizio A. Diaz · 2025
Read Learned Structure in Cartridges : Keys as Shareable Routers in Self-Studied Representations by Maurizio A. Diaz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17032.txt.
Introduction
Maurizio Diaz studies the inner structure of a compressed memory used by large language models. Long inputs make the model's cache of past tokens grow without bound. A method called Cartridges trains a far smaller cache for a text ahead of time. Diaz asks what the trained cache learns, part by part. He tests the claim that its keys act as routers that fetch stored facts.
Tests across tasks and model sizes back the routing claim. Keys can be swapped between tasks with little loss in quality. Most of the learned squeeze sits in the value vectors, not the keys. Diaz also adds a start method that seeds the cache from sampled chunks of text. That start makes training settle faster than past reports showed.
Catalog record
| CREATOR | Maurizio A. Diaz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learned Structure in Cartridges : Keys as Shareable Routers in Self-Studied Representations Maurizio A. Diaz Abstract A bottleneck for long-context LLM inference is the linearly growing KV cache. Recent work has proposed Cartridges , an approach which leverages offline compute to train a much smaller KV cache than is typically required for a full document (up to 40x less memory usage at inference time). In this paper, we present the first mechanistic exploration of the learned Cartridge key-value cache structure. In particular, we propose that (1) Cartridge keys act as stable, shareable retrieval routers for the compressed corpora and (2) most of the learned compression occurs within the Cartridge value vectors. We present empirical evidence of our routing theory across tasks, model families, and model sizes; for example, we can ablate the learned Cartridge key vectors between tasks with little performance loss. Finally, we propose a slight improvement in initialization called Sampled Chunk Initialization (SCI). We suggest that SCI can lead to faster Cartridge convergence than previously demonstrated in the literature. Our findings lay the groundwork for broader empirical study of Cartridge training optimization which may be crucial for further scaling. \workshoptitle Workshop on Mechanistic Interpretability 1 Introduction As context windows grow for large language models (LLMs) vaswani2017attention , users expect to process larger corpora. Common use-cases are large code bases nam2024using , financial filings and market data islam2023financebench , legislature guha2023legalbench , or personal files arora2022can . The most common solution is in-context learning (ICL) wherein we provide the full context to the downstream model; however, ICL-based inference relies on a key-value (KV) cache that grows linearly with context window utilization dong2022survey . Because the cache grows linearly, we need to consider (1) the memory consumption of storing context for ICL and (2) the resulting reduction in throughput due to increased pressure on-device bandwidth and compute fu2024challenges . This leaves users with an undesirable tradeoff between convenience and inference cost. Recent work tackling long-context inference have primarily focused on server-level optimizations. Server-level approaches most notably include prompt (or prefix) caching, prefix-sharing (cascade and bifurcated attention, Hydragen juravsky2024hydragen cascade-inference athiwaratkun2024bifurcated ), and distributed approaches like ring attention. liu2023ring However, these methods do not directly address compute and bandwidth pressures. In response, a growing roster of cache slimming protocols has emerged and we can divide them into two classes: token reducers and key-value compressors. Token reducers rely on summarization, chunking, or filtering to reduce corpora directly via heuristics or natural language jiang2023llmlingua chuang2024learning . Key-value compressors directly leverage sparsity or query oracles to project KV caches into lower rank spaces ge2023context tang2024quest . While these approaches are promising, they typically come with a quality tradeoff. A more recent key-value compression method proposes a two-stage pipeline eyuboglu2025cartridges : 1. Self-Study: Exhaustively chunk the corpus and prompt a model to quiz itself on the content. These synthetic rollouts can be massively parallel and help build a diverse training dataset. 2. Context Distillation: Once we have our conversational traces, we initialize the model with a fixed-size key-value cache of length p\ll d_{\mathcal{C}} where \mathcal{C} is our Corpus. Then, we can train on a context distillation objective to align our Cartridge -initialized model’s next token distribution with the Self-Study synthetic traces hinton2015distilling snell2022learning . Figure 1 : (Left) Cartridges learn to compress long-context documents by first generating synthetic conversations about the corpus and then training a small KV cache using the synthetic traces. This process is called Self-Study . Leveraging a context distillation objective, we back-propagate Self-Study traces into trainable KV caches while keeping the rest of the model frozen. (Right) Here we plot the layer-wise mean singular value spectra of a Llama 3.1 8B Cartridge ’s KV vectors before and after training llama3 . The resulting key vectors are stable while the learned value vectors increase in singular value, representing a more efficient use of representation space due to compression. The synthetic trace generation process is Self-Study and the final context-distilled cache is called a Cartridge . Cartridges exhibit interesting properties, including composability, high recall accuracy on benchmarks, and a tensor structure that is ideal for high throughput prefix-sharing inference engines e.g. Tokasaurus juravsky2025tokasaurus . In this paper, we contribute a preliminary investigation into the mechanisms of a trained Cartridge . In particular, we study the structure of the learned Cartridge key and value vectors across model families, scale, and corpora. We can summarize our results as follows: • We observe that keys barely change during Cartridge training whereas values change significantly. We use singular value analysis to summarize this trend and argue that value vectors learn to use their representation dimensionality more effectively to maximize compression. This result likely explains why random initialization failed to converge in the original Cartridges paper. • We ran an ablation experiment where we train two Cartridges with shared initializers on different tasks and then swap the learned key vectors but not the values. We note that responses remain coherent and the resulting performance loss is mild, which aligns with our first observation. • We investigate a simple but effective initialization scheme for our Cartridge . Rather than using the first p tokens of our corpus to initialize the Cartridge (the main method used in the original paper), we randomly sample chunks throughout the full corpus to maximize structural diversity. This approach leads to statistically significant improvements for training convergence. 2 Preliminaries In this section, we position Cartridges as a new parameter efficient fine-tuning (PeFT) technique that borrows directly from prefix-tuning. We briefly summarize prior work on PeFT mechanistic interpretability, highlighting the lack of related work to prefix-tuning based PeFT methods. Next, we provide more details on Cartridges with a focus on the relevant experimental parameters and notation for our downstream experiments. Finally, we briefly cover the relevant datasets and benchmarks to reduce friction when framing our analyses in Sections 3 and 4. 2.1 Related Work There is a significant body of work exploring task-specific LLM specialization. Common PeFT techniques are low-rank adapters (LoRA) and prefix-tuning, with the former being the most popular method hu2022lora li2021prefix . Cartridges primarily borrow from prefix-tuning: both methods prepend a learned representation to the input sequence and rely on the model to treat the trained prefix as real tokens. However, despite the widespread popularity of PeFT, there is comparatively little research focused on interpreting how LoRAs or prefix-tuning change models at a mechanistic level. Interpreting LoRA Recent interpretability work focuses on LoRA-driven mechanistic changes when tuning for narrowly scoped tasks. For example, Lee et al. studied sparse neural activations during LoRA for nuclear safety applications, while Nijasure et al. found that middle layers (5-15) and specific MLP projections drive ranking performance lee2025mecha nijasure2025relevance . These works focus on neuron-level changes and layer contributions, whereas our analysis examines the geometric structure and transferability of learned representations. Interpreting Prefix-Tuning Prefix-tuning, which is much less popular than LoRA in practice, has even less relevant interpretability work. However, Petrov et al. provide a theoretical analysis of when prefix-tuning works well petrov2024when . They theoretically argue that prefix-tuning "cannot change the relative attention pattern over the content" and we find that this theory aligns strongly with our experimental findings. More concretely, they argue that key vectors have