Tensor Cache: Eviction-conditioned Associative Memory for Transformers
Kabir Swain, Sijie Han, Daniel Karl I. Weidele, Mauro Martino, Antonio Torralba · 2026
Read Tensor Cache: Eviction-conditioned Associative Memory for Transformers by Kabir Swain, Sijie Han, Daniel Karl I. Weidele, Mauro Martino, Antonio Torralba (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22884.txt.
Introduction
Swain and four colleagues propose Tensor Cache, a two-level memory for Transformers that rescues tokens evicted from a sliding window instead of discarding them. Recent tokens stay in exact local attention, while evicted key-value pairs compress into a fixed-size outer-product memory read by later queries. A learned gate fuses both levels. The design improves the memory-quality frontier against bounded-state rivals, and fixes a training shortcut that silently added spurious cross-token products.
Catalog record
| CREATOR | Kabir Swain, Sijie Han, Daniel Karl I. Weidele, Mauro Martino, Antonio Torralba |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46693 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22884
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tensor Cache: Eviction-conditioned Associative Memory for Transformers Kabir Swain, Sijie Han, Daniel Karl I. Weidele, Mauro Martino, Antonio Torralba Abstract Autoregressive Transformer KV caches grow linearly with context length; sliding-window caching bounds memory but discards evicted tokens entirely, so relevant evidence outside the window becomes inaccessible. We introduce Tensor Cache , a two-level cache that pairs sliding-window softmax attention as a first-level cache (L1) with a fixed-size outer-product fast-weight memory as a second-level cache (L2) fed by KV pairs evicted from the window. Recent tokens remain in exact local attention; evicted pairs are compressed into a per-layer matrix A A and read by future queries through a single matrix multiplication, exploiting the linear-attention identity q t ( k i ⊗ v i ) = ⟨ q t , k i ⟩ v i q_{t}(k_{i}\otimes v_{i})=\langle q_{t},k_{i}\rangle v_{i} . A learned scalar gate fuses the L1 and L2 outputs, and per-head decay and write-rate parameters are trained end-to-end. The outer-product memory and the read identity are well-known; our contribution is their use as an L2 cache fed exclusively by sliding-window evictions, plus identifying that the common chunked-mean training shortcut A ← λ A + η ( k ¯ ⊗ v ¯ ) A\!\leftarrow\!\lambda A\!+\!\eta(\bar{k}\!\otimes\!\bar{v}) silently introduces C 2 − C C^{2}{-}C spurious cross-token outer products per chunk, and closing the gap with a parallel weighted-sum scan equivalent to per-token writes within float32 epsilon. Across systems scaling, controlled associative recall, long-context language modeling, and memory-capacity diagnostics, Tensor Cache improves the memory–quality frontier over bounded-state baselines. Figure 1 : Tensor Cache. Each layer keeps a local KV ring buffer (L1) and a fixed-size memory A (L2). On eviction, (k,v) is written into A via an outer-product update; queries read both paths and fuse via a learned gate. 1 Introduction Autoregressive Transformer inference caches per-layer keys and values (KV) so the prefix is not recomputed at every step ( 26 ; 23 ) , but retained KV state grows linearly with context length, depth, head dimension, and serving concurrency ( 14 ) . Sliding-window caching bounds this cost by keeping only the most recent W tokens ( 29 ) , at the price of hard forgetting : once a token leaves the window, its key and value are removed from attention entirely. We introduce Tensor Cache (TC) , a bounded-state mechanism that turns KV eviction from a deletion event into an associative-memory write. Each layer keeps exact softmax attention over the recent window (a first-level cache, L1), and when the ring buffer overwrites an old entry, the evicted pair (k_{w},v_{w}) is written into a fixed-size outer-product matrix A (a second-level cache, L2) of the kind used in fast-weight programmers ( 22 ) and modern linear-attention variants ( 21 ; 13 ; 2 ; 18 ) . Future queries read this state by q_{t}A\;\approx\;\sum_{i}\alpha_{i}\langle q_{t},k_{i}\rangle v_{i}, i.e. the same query–key alignment used by attention also addresses the compressed past, but without retaining the individual evicted entries. A learned scalar gate fuses the L1 and L2 outputs. The outer-product memory and read identity q_{t}(k_{i}\otimes v_{i})=\langle q_{t},k_{i}\rangle v_{i} are not new. Our contribution is to use them as an L2 cache fed exclusively by sliding-window evictions, rather than as a parallel running average over every token (mLSTM, Infini-attention) or as a wholesale replacement for attention (RetNet, Mamba) ( 25 ; 12 ) . We additionally identify a previously-overlooked train-time shortcut: a naive chunked-scan implementation summarizes each chunk by its mean and writes A\leftarrow\lambda A+\eta(\bar{k}\otimes\bar{v}) , which differs from the per-token sum by C^{2}-C cross-token outer products k_{i}\otimes v_{j} ( i\neq j ) that never appear at inference. We close this gap with a parallel weighted-sum scan that is mathematically equivalent to per-token writes for the outer rule, verified to float32 epsilon. The parallel-scan formulation itself is standard in the linear-attention literature ( 2 ; 18 ) ; we make its applicability to eviction-conditioned memory explicit and quantify the cost of skipping it. TC is not a lossless replacement for full attention. Full KV remains the exact-content baseline when affordable; in evaluations beyond the training block size, Full-KV results should be read as exact-content but positionally extrapolated ( 24 ; 5 ; 19 ) . TC targets the regime where full-KV state is too expensive but pure sliding-window decoding forgets too aggressively. Contributions. • A two-level cache architecture combining sliding-window softmax attention (L1) with an outer-product fast-weight memory (L2) fed exclusively by KV pairs evicted from the window. Unlike prior outer-product memories that ingest every token, our eviction-conditioned write makes the matrix specifically a second-level store for what local attention can no longer reach. • Identification and closure of a chunked-mean training shortcut. The shortcut introduces C^{2}{-}C spurious cross-token outer products per chunk; the parallel weighted-sum scan removes them at the same compute cost (verified within 10^{-7} relative error). • Empirical evaluation across systems scaling, real-text long-context language modeling, controlled associative recall, raw memory capacity, and ablations. Tensor Cache attains 100% matched-gap synthetic recall while using 72–84% less inference state than Full KV, and on OpenWebText long-context language modeling it attains the lowest mean NLL of any tested method—including Full KV, Window KV, StreamingLLM, and Infini-attention—at every evaluation context length from 1024 to 32{,}768 tokens with the single exception of L=2048 , where it is statistically indistinguishable from Infini-attention; at 32\times the trained context Tensor Cache reaches NLL 5.14 versus 6.00 for Full KV at 2.4\times greater peak GPU memory. 2 Related Work KV cache systems and eviction. KV-cache footprint scales with context length and serving concurrency. Systems work such as PagedAttention ( 14 ) and CacheGen ( 17 ) improves allocation, compression, or movement, but the logical retained state still scales with the amount of context kept. A complementary line reduces KV size by retaining only selected tokens: sliding-window attention (Longformer ( 3 ) , StreamingLLM with attention sinks ( 29 ) ) and importance-based selection (H 2 O, SnapKV, CAOTE ( 31 ; 15 ; 10 ) ). These methods decide which discrete entries to keep. Tensor Cache instead accepts FIFO eviction and converts the evicted pair into a compact associative-memory update. Memory-augmented and recurrent Transformers. Transformer-XL reuses hidden states across segments ( 6 ) , Compressive Transformers compress older activations ( 20 ) , Memorizing Transformers use external kNN memory ( 28 ) , and Recurrent Memory Transformer carries information through memory tokens ( 4 ) . Most relevant to bounded-state decoding, LESS combines sparse KV retention with a low-rank recurrent cache ( 9 ) , and Infini-attention combines masked local attention with a bounded compressive memory updated every token through a normalized outer-product write ( 18 ) . Tensor Cache differs in the semantics of the write: a key–value pair is first used in exact local softmax attention, and only when the ring buffer overwrites it is the pair written into the auxiliary state. Fast weights, outer-product memories, and linear attention. The outer-product fast-weight memory traces back to Schmidhuber’s fast-weight programmer ( 22 ) ; 1 revisited fast weights as short-term memory. Widrow–Hoff ( 27 ) introduced the error-correcting principle behind the delta rule. 21 reframed linear attention ( 13 ) as a fast-weight programmer with accumulated k\otimes v writes,