Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models

Haoyu Wang, Peihao Wang, Mufei Li, Shikun Liu, Siqi Miao, Zhangyang Wang, Pan Li · 2025

Read Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models by Haoyu Wang, Peihao Wang, Mufei Li, Shikun Liu, Siqi Miao, Zhangyang Wang, Pan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07334.txt.

Description

Haoyu Wang and six colleagues give large language models a way to respect the graph shape of their input. Models read text as one flat line, token after token. That flattening hides links between chunks, which hurts tasks built on connected records. Their method, Graph-KV, stores each chunk's inner state as a compact cache.

A target chunk then attends only to the caches of its named source chunks. This carves attention into a graph-shaped mask, like messages passing along edges. Careful placement of position codes cuts bias and saves window space. The team tested the method on seven retrieval benchmarks and a new task drawn from science papers and their citation webs. It also classified papers inside a citation network. Across the tests, Graph-KV beat plain serial reading and other strong baselines.

Catalog record

CREATORHaoyu Wang, Peihao Wang, Mufei Li, Shikun Liu, Siqi Miao, Zhangyang Wang, Pan Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07334

Browse the full library · About the Tower of Babel