S 3 -Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference

Qingsen Ma, Dianyun Wang, Yaoye Wang, Lechen Ning, Sujie Zhu, Xiaohang Zhang, Jiaming Lyu, Linhao Ren, Zhenbo Xu, Zhaofeng He · 2026

Read S 3 -Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference by Qingsen Ma, Dianyun Wang, Yaoye Wang, Lechen Ning, Sujie Zhu, Xiaohang Zhang, Jiaming Lyu, Linhao Ren, Zhenbo Xu, Zhaofeng He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17702.txt.

Description

Qingsen Ma and co-authors present S3-Attention for long-context inference in language models. Full context makes the KV cache grow in a linear way. Offloading to outside retrievers saves memory. But retrieved passages can match words while missing the true causal thread. S3-Attention turns memory-bound inference into a streaming task. Retrieval is endogenous, drawn from the model's own attention.

Attention-aligned picks should track what the model truly needs. That keeps long reads exact on a fixed memory budget. Book and code length inputs are the target use. A streaming design also fits live chat with long histories. The work joins memory and attention in one frame.

Catalog record

CREATORQingsen Ma, Dianyun Wang, Yaoye Wang, Lechen Ning, Sujie Zhu, Xiaohang Zhang, Jiaming Lyu, Linhao Ren, Zhenbo Xu, Zhaofeng He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17702

Browse the full library · About the Tower of Babel