Slash the Sink: Sharpening Structural Attention Inside LLMs
Yiming Liu, Bin Lu*, Xinbing Wang, Chenghu Zhou, Meng Jin* · 2026
Read Slash the Sink: Sharpening Structural Attention Inside LLMs by Yiming Liu, Bin Lu*, Xinbing Wang, Chenghu Zhou, Meng Jin* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10503.txt.
Description
Yiming Liu and four colleagues show language models already grasp graph structure internally, then sharpen it. Large language models show remarkable semantic grasp but struggle with structural grasp on flattened graph layouts. Existing solutions train external graph adapters or fine-tune, at high cost. Their 2026 paper investigates how language models work inside.
The authors find models reconstruct graph layout on their own, shown by a sawtooth pattern in attention maps matching the token-level adjacency matrix. That intrinsic grasp is diluted by the attention sink, where models give outsized attention to initial tokens. The authors formalize this dilution as a coding bottleneck from a conflict: anisotropic bias essential for language tasks suppresses layout-aware local pooling needed for graph reasoning. Their training-free solution, Slash, amplifies internal structural grasp via plug-and-play attention reshaping. Tests on graph tasks and molecular prediction show large, steady gains across models.
Catalog record
| CREATOR | Yiming Liu, Bin Lu*, Xinbing Wang, Chenghu Zhou, Meng Jin* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10503