Interdomain Attention: Beyond Token-Level Key-Value Memory
Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin, Zhuo Sun, Wenlong Chen, Samir Bhatt, Yingzhen Li · 2026
Read Interdomain Attention: Beyond Token-Level Key-Value Memory by Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin, Zhuo Sun, Wenlong Chen, Samir Bhatt, Yingzhen Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24330.txt.
Introduction
Kiyohara and six co-authors propose Interdomain Attention, mixing attention with recurrent memory in one layer instead of stacking separate layers. Viewing attention as kernel regression, the design squeezes key features and values into recurrent memory run by a state-space model. It rebuilds attention values from memory states. The design keeps the memory model's speed edge. Text-trained models from 125 million to 1.3 billion parameters beat an S4D baseline at equal memory size. On genome data with a Caduceus base, it matches or beats the original.
Catalog record
| CREATOR | Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin, Zhuo Sun, Wenlong Chen, Samir Bhatt, Yingzhen Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54419 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interdomain Attention: Beyond Token-Level Key-Value Memory Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin, Zhuo Sun, Wenlong Chen, Samir Bhatt, Yingzhen Li Abstract Transformers and deep state space models (SSMs) sit at opposite ends of a basic design choice: attention routes each query through a growing key-value (KV) cache by content-based matching at quadratic cost, while deep SSMs compress context into a fixed-size recurrent state that is not directly addressed by query-key matching. We propose Interdomain Attention , which integrates an SSM into an attention module through kernel methods: an attention kernel is approximated by a finite feature map, the resulting key features and values are projected onto a shared set of basis functions maintained by a single SSM recurrence, and each query attends to the compressed coefficients through its own feature map, recovering query-conditioned attention over a fixed-size state. The scalable layer is a learned relaxation of this derivation, and we validate its components through ablations. In a 125 M–1.3 B autoregressive language-modeling study on FineWeb-Edu at matched recurrent-state budget, Interdomain Attention improves on an SSM token mixer at every scale, surpasses a same-recipe softmax baseline at 1.3 B on validation perplexity and on the eight-task commonsense suite, and inherits the length-flat behavior of its fixed-state core out to 3.5 × 3.5\times the training context. Ablations indicate that the query-conditioned projection is the main source of the gain. † † affiliationnotice: Equal contribution 1 Introduction Softmax attention and state space models (SSMs) sit at opposite ends of a basic design choice for sequence models. Attention ( 59 ) keeps a per-token key-value cache and lets each query route through it by content-based matching, which gives sharp recall but costs \mathcal{O}(N_{q}N) work and an \mathcal{O}(N) KV state, where N_{q} is the query length and N is the key-value length. Hardware-aware kernels ( 14 ; 16 ) , KV caching ( 49 ) , and distributed sharding ( 51 ) mitigate but do not remove this scaling. Deep SSMs, building on HiPPO ( 21 ) and realized by S4 ( 24 ) , S4D ( 23 ) , and Mamba ( 22 ; 15 ) , take the opposite stance: they compress the entire context into a fixed-size recurrent state that updates in \mathcal{O}(1) time per step. The cost of this compression is that the fixed state is not directly addressed by query-key matching. Hybrid architectures interleave the fine-grained, content-based retrieval of attention with the efficient long-range compression of SSMs ( 34 ; 20 ; 46 ; 18 ; 19 ; 7 ) , and a growing line of linear and sub-quadratic attention reduces the cost of attention directly ( 30 ; 41 ; 42 ; 40 ; 54 ; 63 ; 64 ) . We take a different route: rather than stacking a recurrent layer next to attention, we ask whether an SSM can keep the efficiency of a fixed-size state while still allowing each query to attend to the compressed history. Holding the recurrent core to S4D and the per-token recurrent state to a fixed budget, we study what closes the gap between an S4D token mixer and a query-conditioned mixer. We answer this with Interdomain Attention , a token mixer in which keys and values are mapped to a shared SSM basis by a single complex S4D recurrence. At each position, the query attends to these compressed coefficients instead of attending to every past token. The construction is motivated by representing an attention kernel through a finite feature map and projecting the key features onto HiPPO basis functions, which yields a fixed-size state independent of sequence length. The scalable implementation is not a literal realization of the kernel derivation: it uses a learned SiLU/ \ell_{2} feature map, input normalization, and a denominator-free readout. We therefore use the derivation as design motivation and evaluate the resulting layer empirically through ablations. Our contributions are: • A construction of a query-conditioned fixed-state token mixer from a kernel-regression view of attention and a HiPPO-style basis projection, with an explicit boundary between the ideal derivation and the scalable implementation ( Sections 3.1 and 3.3 ). • A mechanism decomposition at 125 M parameters that separates the contributions of dual key/value input and query-conditioned projection, and identifies the projection as the dominant axis (Appendix B.1 ). • A 125 M–1.3 B iso-state language-modeling study on FineWeb-Edu in which Interdomain Attention improves over an S4D token mixer at every scale, surpasses a same-recipe softmax baseline on validation perplexity and the eight-task commonsense suite at 1.3 B, and preserves the length-flat behavior of the fixed-state core to 3.5\times the training context ( Figure 2 , Appendix B.4 , Table 2 ). 2 Background In this section, we briefly review attention mechanisms and state space models. Attention as Kernel Regression. Standard dot-product attention ( 59 ) maps input tokens x_{n}\in\mathbb{R}^{d} to queries, keys, and values via q_{n}=W_{q}x_{n}\in\mathbb{R}^{d} , k_{n}=W_{k}x_{n}\in\mathbb{R}^{d} , v_{n}=W_{v}x_{n}\in\mathbb{R}^{d} , and computes the output for the i -th token as \displaystyle o_{i}=\frac{\sum_{n=1}^{N}\mathcal{K}(q_{i},k_{n})\,v_{n}}{\sum_{n^{\prime}=1}^{N}\mathcal{K}(q_{i},k_{n^{\prime}})}, (1) where \mathcal{K}(q,k)=\exp(q^{\top}k/\sqrt{d}) for softmax attention. This is one realization of a Nadaraya-Watson kernel regression estimator ( 37 ; 61 ) , a connection noted in several works on kernel attention ( 57 ; 30 ; 11 ) . This view makes the choice of kernel \mathcal{K} a design degree of freedom: replacing the softmax kernel with one that admits a finite or learned feature representation enables sub-quadratic computation and memory. Attention then performs non-parametric regression over the values v_{n} at test time, with \mathcal{K} determining the weighting over the context, connecting to the broader test-time regression ( 60 ) or memorization (e.g. Titans; 4 ). State Space Models and HiPPO. A linear state space model maps an input signal z(t)\in\mathbb{R} to a latent state u(t)\in\mathbb{R}^{M} via: \displaystyle\dot{u}(t)=A(t)\,u(t)+B(t)\,z(t). (2) HiPPO ( 21 ) gives initializations for A(t)\in\mathbb{R}^{M\times M} and B(t)\in\mathbb{R}^{M} under which the state u(t) maintains optimal projections of the input history onto M time-varying orthogonal basis functions \{\phi_{m}^{(t)}\}_{m=1}^{M} . This forms the foundation for deep SSM architectures such as S4 ( 24 ) , S5 ( 52 ) , S4D ( 23 ) , and Mamba ( 22 ; 15 ) . 3 Interdomain Attention (a) Standard attention Q K V {\color[rgb]{0.6,0,0}A}=\operatorname{softmax}\!\!\left(\dfrac{{\color[rgb]{0.1172,0.1953,0.7852}Q}{\color[rgb]{0.7852,0.4297,0}K}^{\top}}{\sqrt{d}}\right) N_{q}\times N scores value aggregation {\color[rgb]{0.5,0.5,0.5}\underbrace{{\color[rgb]{0.6,0,0}A}}_{N_{q}\times N}}\cdot{\color[rgb]{0.5,0.5,0.5}\underbrace{{\color[rgb]{0.5898,0.1563,0.668}V}}_{N\times d}} Output (b) Interdomain attention Q K V \xi_{\omega} \xi_{\omega} SSM recurrence kernel query {\color[rgb]{0.1172,0.1953,0.7852}F_{q}} interdomain states ({\color[rgb]{0.7852,0.4297,0}U},\;{\color[rgb]{0.5898,0.1563,0.668}\Gamma},\;\eta) state-space readout {\color[rgb]{0.5,0.5,0.5}\underbrace{{\color[rgb]{0,0,0}\dfrac{{\color[rgb]{0.1172,0.1953,0.7852}F_{q}}\;{\color[rgb]{0.7852,0.4297,0}U}^{\top}}{{\color[rgb]{0.1172,0.1953,0.7852}F_{q}}\;{\color[rgb]{0.7852,0.4297,0}U}^{\top}\eta}}}_{N_{q}\times M}}\;\cdot\;{\color[rgb]{0.5,0.5,0.5}\underbrace{{\color[rgb]{0.5898,0.1563,0.668}\Gamma}}_{M\times d}} Output Figure 1 : (a) Standard attention computes N_{q}\times N scores from Q and K , then multiplies by V to produce the output. (b) Interdomain attention maps queries and keys into a shared feature space to produce the kernel query matrix {\color[rgb]{0.1172,0.1953,0.7852}F_{q}} ( N_{q}\times R ). Keys and values are compressed via SSM recurrence into M interdomain states: {\color[rgb]{0.7852,0.4297,0}U} ( M\times R , key feature projections), {\color[rgb]{0.5898,0.1563,0.668}\Gamma} ( M\times d , value projections), and \eta ( M\times 1 , normalizing constants). State-space readout computes the output via the N_{q}\times M cross-covariance {\color[rgb]{0.1172,0.1953,0.7852}F_{q}}{\color[rgb]{0.7852,0.4297,0}U}^{\top} . The kernel-regression view suggests a direct way to make attention recurrent: summarize the keys and values in a fixed-size set of basis coefficients, then let each query attend to those coefficients. HiPPO-style SSMs provide a natural online mechanism for maintaining such coefficients. Building on the