S 3 -Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference
Qingsen Ma, Dianyun Wang, Yaoye Wang, Lechen Ning, Sujie Zhu, Xiaohang Zhang, Jiaming Lyu, Linhao Ren, Zhenbo Xu, Zhaofeng He · 2026
Read S 3 -Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference by Qingsen Ma, Dianyun Wang, Yaoye Wang, Lechen Ning, Sujie Zhu, Xiaohang Zhang, Jiaming Lyu, Linhao Ren, Zhenbo Xu, Zhaofeng He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17702.txt.
Description
Qingsen Ma and co-authors present S3-Attention for long-context inference in language models. Full context makes the KV cache grow in a linear way. Offloading to outside retrievers saves memory. But retrieved passages can match words while missing the true causal thread. S3-Attention turns memory-bound inference into a streaming task. Retrieval is endogenous, drawn from the model's own attention.
Attention-aligned picks should track what the model truly needs. That keeps long reads exact on a fixed memory budget. Book and code length inputs are the target use. A streaming design also fits live chat with long histories. The work joins memory and attention in one frame.
Catalog record
| CREATOR | Qingsen Ma, Dianyun Wang, Yaoye Wang, Lechen Ning, Sujie Zhu, Xiaohang Zhang, Jiaming Lyu, Linhao Ren, Zhenbo Xu, Zhaofeng He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17702