CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

Sayak Dutta · 2026

Read CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention by Sayak Dutta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27229.txt.

Introduction

Dutta introduces CARVE, a recurrent language model design that lets the model peek at its own memory before forgetting. Recurrent models squeeze everything that came before into one fixed-size state matrix. Leading delta-rule models gate that state using only the incoming token, so erasure decisions are memory-blind. In 2026, value-axis coupling blocks the fast triangular chunk solver that makes recurrent training efficient.

Restricting all gating to the key axis restores that single fast solve unchanged. Content-aware erasing reuses the output tensor the kernel already writes, taking its chunk mean as a cheap memory glimpse. A zero-start projection feeds that glimpse to the erase gate, so training begins identical to the baseline. Selectivity grows as training proceeds. Value efficiency trims the write gate to a lean per-head form. Speed and memory awareness improve together.

Catalog record

CREATORSayak Dutta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27229

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention Sayak Dutta Abstract What if a recurrent language model could look at its own memory before deciding what to forget—at zero extra cost? Recurrent sequence models in the delta-rule family maintain a fixed-size state matrix 𝑺 t ∈ ℝ d v × d k \bm{S}_{t}\in\mathbb{R}^{d_{v}\times d_{k}} that compresses all past context into d v ​ d k d_{v}d_{k} numbers. The current state of the art [ 12 ] equips this update with element-wise matrix gates: a full d v × d k d_{v}{\times}d_{k} erase mask and a full d v × d k d_{v}{\times}d_{k} write mask. The result is powerful but carries two intrinsic defects. First, both gates are computed solely from the incoming token, making the model memory-blind : it must decide what to erase without observing what it has already stored. Second, the d v d_{v} -dimensional write gate projection carries the same parameter count and memory footprint as the value projection itself—a bandwidth cost with negligible expressive return over a single scalar per head. Worse, the value-axis coupling in the erase gate mathematically prevents the model from using the WY-form triangular chunk solver that is the engine of efficient recurrent training: the intra-chunk system decomposes into d v d_{v} independent solves instead of one, collapsing throughput to serial-recurrence cost (Theorem 11 ). We introduce CARVE ( C ontent- A ware R ecurrent with V alue E fficiency), which resolves all three problems at once. The key observation is architectural: by restricting all gating to the key axis only, the intra-chunk coupling matrix becomes independent of the value index, restoring the single WY-form triangular solve unmodified. Within this key-axis constraint, CARVE introduces two innovations. Content-aware erase via output reuse. Rather than reading the state matrix 𝑺 t − 1 \bm{S}_{t-1} from HBM (which would double memory traffic), CARVE re-uses the recurrent output tensor 𝒐 c , ⋅ = 𝑺 ​ 𝒒 c , ⋅ \bm{o}_{c,\cdot}=\bm{S}\,\bm{q}_{c,\cdot} that the kernel must write to HBM regardless, computing its chunk mean 𝒎 c \bm{m}_{c} at negligible cost. This single signal, passed through a zero-initialised low-rank projection 𝑼 b \bm{U}_{b} , gives the erase gate its first glimpse of stored content. At initialisation, 𝑼 b = 𝟎 \bm{U}_{b}{=}\mathbf{0} and CARVE is bit-identical to the baseline; as training proceeds the gate activates, introducing memory-conditioned selectivity. We prove that the one-chunk staleness of 𝒎 c \bm{m}_{c} induces a gate perturbation of only 𝒪 ⁡ ( 1 / L ) \mathcal{O}(1/\sqrt{L}) , which is why the measured deviation from exact gating is flat at 0.18 % \mathbf{0.18\%} across all chunk lengths up to L = 128 L{=}128 . Scalar value write gate. CARVE replaces the d v d_{v} -dimensional per-value write-gate projection with a single scalar w h , t w_{h,t} per head. This is a d v d_{v} -fold parameter reduction—from 589,824 589{,}824 to 9,216 9{,}216 write-gate parameters per layer at H = 12 H{=}12 , d v = 768 d_{v}{=}768 —and we prove it is lossless for single-slot associative recall (Theorem 15 ). At the 1.3 1.3 B-parameter scale trained on 100 100 B tokens of FineWeb-Edu on NVIDIA H100—three-seed averages against the best prior recurrent baseline—CARVE delivers improvements on every front simultaneously. On WikiText language modelling it reaches perplexity 15.72 \mathbf{15.72} versus 15.90 15.90 , a − 0.18 -0.18 reduction that constitutes a 4.5 ​ σ \mathbf{4.5\sigma} effect across seeds; the hybrid variant extends this lead to 15.41 15.41 versus 15.62 15.62 . Across nine common-sense reasoning benchmarks CARVE leads every recurrent model by + 0.63 \mathbf{+0.63} pp average zero-shot accuracy, and on RULER in-context retrieval probes it sets the state of the art on every S-NIAH and MK-NIAH context length and tops all six real-world recall benchmarks. Remarkably, none of this comes at a hardware cost: throughput is within 0.4 % \mathbf{0.4\%} of the matrix-gated baseline (within measurement noise), peak memory is − 𝟏𝟑 % \mathbf{-13\%} lower, and mixer parameters are − 𝟏𝟗 % \mathbf{-19\%} fewer. CARVE is the first architecture in the gated delta family to simultaneously achieve content-aware gating, full WY-form chunk-parallel training, and reduced parameter count—all with a provably bounded approximation and six formal theoretical guarantees. 1 Introduction The memory bottleneck in sequence modelling. Every practical language model ultimately confronts the same trade-off: how much context should be remembered, and at what hardware cost? Transformers resolve this by keeping everything—an exact, quadratic record of every past token [ 37 ] . This is spectacularly powerful but spectacularly expensive: training a 1\text{B} -parameter Transformer on sequences of length T{=}8192 requires \mathcal{O}(T^{2}d) memory accesses for the attention block alone, and inference latency grows linearly with the KV-cache. Systems engineers have pushed back heroically—FlashAttention [ 5 ] , multi-query attention, sliding-window hybrids—but no IO-tiling removes the fundamental \mathcal{O}(T^{2}) compute from full attention. At deployment scale, where models serve millions of long-context requests, these costs are not engineering inconveniences; they are hard limits on what is economically feasible. The return of recurrence. Recurrent architectures offer a fundamentally different bargain: compress all past context into a fixed-size state matrix \bm{S}_{t}\in\mathbb{R}^{d_{v}\times d_{k}} , then update it in \mathcal{O}(d_{v}d_{k}) operations per token. Inference becomes constant-memory and constant-cost regardless of sequence length—a property that scales directly into practical advantages at deployment. The price paid is compression: \bm{S}_{t} has d_{v}d_{k} real-valued entries, and writing a new token must inevitably overwrite older entries, requiring a principled forgetting mechanism. Early recurrences (LSTMs, GRUs) relied on learned scalar gates but struggled with long-range information flow. A newer family—linear recurrences [ 11 ] , SSMs [ 10 ; 6 ] —exchanged full expressivity for hardware efficiency via diagonal state matrices, enabling fast parallel scans but constraining the set of associations the state can represent. The delta rule [ 38 ; 31 ; 39 ] takes a more principled path. It maintains a full matrix state \bm{S}_{t}\in\mathbb{R}^{d_{v}\times d_{k}} and updates it via an associative-memory error-correction law, \bm{S}_{t}=\bm{S}_{t-1}+\underbrace{(\bm{v}_{t}-\bm{S}_{t-1}\bm{k}_{t})}_{\text{prediction error}}\bm{k}_{t}^{\top}, (1) where \bm{q}_{t},\bm{k}_{t}\in\mathbb{R}^{d_{k}} are query and key vectors, \bm{v}_{t}\in\mathbb{R}^{d_{v}} is the value, and \bm{S}_{t-1}\bm{k}_{t} is the memory’s current prediction for the value stored at key \bm{k}_{t} . The update writes a rank-1 correction precisely proportional to the prediction error—the Hebbian update of a Hopfield network [ 27 ] with error-corrective feedback. GDN [ 39 ] adds a scalar forgetting gate; GDN-2 [ 12 ] replaces the scalar with a full d_{v}{\times}d_{k} matrix gate, yielding the state update \bm{S}_{t}=(\bm{1}-\bm{B}_{t})\odot\bm{S}_{t-1}+\bm{W}_{t}\odot\tilde{\Delta}_{t},\qquad\tilde{\Delta}_{t}=(\bm{v}_{t}-\bm{S}_{t-1}\bm{k}_{t})\bm{k}_{t}^{\top}, (2) where \bm{B}_{t},\bm{W}_{t}\in(0,1)^{d_{v}\times d_{k}} are element-wise matrix gates, and \tilde{\Delta}_{t} is the outer-product prediction-error correction. GDN-2 achieves compelling downstream performance [ 12 ] , and we treat it as the direct prior-art baseline throughout this paper. Three structural limitations of matrix-gated delta recurrences. Memory-blind gating. Both gates in ( 2 ) are computed as linear projections of the current input token \bm{x}_{t} . The model must decide what fraction of key slot k to erase without ever consulting \bm{S}_{t-1} —without knowing whether slot k holds a high-value association to be protected or stale noise to be cleared. This is content-oblivious forgetting: decisions are made on the basis of what is arriving, not what is already stored. Content-dependent selective forgetting—essential for tasks requiring precise slot management—is structurally impossible in this gating regime. Write-gate bandwidth bottleneck. The \bm{W}_{t} projection

Read the complete text