Order-Level Attention Similarity Across Language Models: A Latent Commonality

Jinglin Liang, Jin Zhong, Shuangping Huang, Yunqing Hu, Huiyuan Zhang, Huifang Li, Lixin Fan, Hanlin Gu · 2025

Read Order-Level Attention Similarity Across Language Models: A Latent Commonality by Jinglin Liang, Jin Zhong, Shuangping Huang, Yunqing Hu, Huiyuan Zhang, Huifang Li, Lixin Fan, Hanlin Gu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05064.txt.

Description

Jinglin Liang and co-authors ask whether language models share hidden patterns in how they gather context. Past work looked at single models or single attention heads, so cross-model traits stayed out of view.

The team defines order-level attention, built by splitting a known attention rollout method by order. They find that this measure lines up across models at the same order, a shared trait they call a latent commonality. That shared pattern could deepen grasp of how models work and may help move knowledge from one model to another in time.

Catalog record

CREATORJinglin Liang, Jin Zhong, Shuangping Huang, Yunqing Hu, Huiyuan Zhang, Huifang Li, Lixin Fan, Hanlin Gu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05064

Browse the full library · About the Tower of Babel