Analogies between Transformer Layers and Power Method
Chenglong Li, Claudio Altafini · 2026
Read Analogies between Transformer Layers and Power Method by Chenglong Li, Claudio Altafini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25619.txt.
Description
Li and Altafini show an analogy between one transformer layer, ignoring its feedforward network, and one step of the power method. Passing through a layer tilts tokens toward the principal eigenvector of a matrix formed by multiplying that layer's output and value weight matrices. When all layers share weights, this alignment grows especially clear in experiments and admits an analytic proof. The analogy further suggests steering transformer output toward any chosen direction in token space.
Catalog record
| CREATOR | Chenglong Li, Claudio Altafini |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 129844 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25619