Decomposing Query-Key Feature Interactions Using Contrastive Covariances
Andrew Lee, Yonatan Belinkov, Fernanda Viégas, Martin Wattenberg · 2026
Read Decomposing Query-Key Feature Interactions Using Contrastive Covariances by Andrew Lee, Yonatan Belinkov, Fernanda Viégas, Martin Wattenberg (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04752.txt.
Description
Andrew Lee and 3 co-authors open up transformer attention in a 2026 paper on query-key interplay. Attention decides which tokens a model looks at, but why one token wins has lacked good tools for explanation.
The authors study the joint space formed by queries and keys, the two vectors whose match yields attention scores. Their contrastive covariance method splits that space into low-rank parts that humans can read. High attention arises when query and key traits align inside those parts. After tests in theory and simplified settings, they apply the method to large language models. It found readable subspaces for semantic groups and binding traits. Attention scores could then be credited to specific traits.
Catalog record
| CREATOR | Andrew Lee, Yonatan Belinkov, Fernanda Viégas, Martin Wattenberg |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04752