Slash the Sink: Sharpening Structural Attention Inside LLMs

Yiming Liu, Bin Lu*, Xinbing Wang, Chenghu Zhou, Meng Jin* · 2026

Read Slash the Sink: Sharpening Structural Attention Inside LLMs by Yiming Liu, Bin Lu*, Xinbing Wang, Chenghu Zhou, Meng Jin* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10503.txt.

Description

Yiming Liu and four colleagues show language models already grasp graph structure internally, then sharpen it. Large language models show remarkable semantic grasp but struggle with structural grasp on flattened graph layouts. Existing solutions train external graph adapters or fine-tune, at high cost. Their 2026 paper investigates how language models work inside.

The authors find models reconstruct graph layout on their own, shown by a sawtooth pattern in attention maps matching the token-level adjacency matrix. That intrinsic grasp is diluted by the attention sink, where models give outsized attention to initial tokens. The authors formalize this dilution as a coding bottleneck from a conflict: anisotropic bias essential for language tasks suppresses layout-aware local pooling needed for graph reasoning. Their training-free solution, Slash, amplifies internal structural grasp via plug-and-play attention reshaping. Tests on graph tasks and molecular prediction show large, steady gains across models.

Catalog record

CREATORYiming Liu, Bin Lu*, Xinbing Wang, Chenghu Zhou, Meng Jin*
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10503

Browse the full library · About the Tower of Babel