SinkTrack: Attention Sink based Context Anchoring for Large Language Models

Xu Liu, Guikun Chen, Wenguan WangThe State Key Lab of Brain-Machine Intelligence, Zhejiang University · 2026

Read SinkTrack: Attention Sink based Context Anchoring for Large Language Models by Xu Liu, Guikun Chen, Wenguan WangThe State Key Lab of Brain-Machine Intelligence, Zhejiang University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10027.txt.

Description

Liu, Chen, and Wang propose SinkTrack, a training-free context anchoring method using the attention sink: the first token draws high attention throughout generation. Treating the first token as an anchor, SinkTrack injects key image or instruction features into its representation, keeping the model anchored to initial context. Experiments show SinkTrack reduces hallucination and context forgetting in text tasks, gaining about 22 percent on SQuAD2 with Llama3-8B-Instruct. In multimodal tasks, it gains about 23 percent on M3CoT with Qwen2-VL-7B-Instruct. Gains hold across architectures and scales.

Catalog record

CREATORXu Liu, Guikun Chen, Wenguan WangThe State Key Lab of Brain-Machine Intelligence, Zhejiang University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50709
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10027

Browse the full library · About the Tower of Babel