SinkTrack: Attention Sink based Context Anchoring for Large Language Models
Xu Liu, Guikun Chen, Wenguan WangThe State Key Lab of Brain-Machine Intelligence, Zhejiang University · 2026
Read SinkTrack: Attention Sink based Context Anchoring for Large Language Models by Xu Liu, Guikun Chen, Wenguan WangThe State Key Lab of Brain-Machine Intelligence, Zhejiang University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10027.txt.
Description
Liu, Chen, and Wang propose SinkTrack, a training-free context anchoring method using the attention sink: the first token draws high attention throughout generation. Treating the first token as an anchor, SinkTrack injects key image or instruction features into its representation, keeping the model anchored to initial context. Experiments show SinkTrack reduces hallucination and context forgetting in text tasks, gaining about 22 percent on SQuAD2 with Llama3-8B-Instruct. In multimodal tasks, it gains about 23 percent on M3CoT with Qwen2-VL-7B-Instruct. Gains hold across architectures and scales.
Catalog record
| CREATOR | Xu Liu, Guikun Chen, Wenguan WangThe State Key Lab of Brain-Machine Intelligence, Zhejiang University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50709 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10027