Analogies between Transformer Layers and Power Method

Chenglong Li, Claudio Altafini · 2026

Read Analogies between Transformer Layers and Power Method by Chenglong Li, Claudio Altafini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25619.txt.

Description

Li and Altafini show an analogy between one transformer layer, ignoring its feedforward network, and one step of the power method. Passing through a layer tilts tokens toward the principal eigenvector of a matrix formed by multiplying that layer's output and value weight matrices. When all layers share weights, this alignment grows especially clear in experiments and admits an analytic proof. The analogy further suggests steering transformer output toward any chosen direction in token space.

Catalog record

CREATORChenglong Li, Claudio Altafini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE129844
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25619

Browse the full library · About the Tower of Babel