Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability
Kingsuk Maitra · 2026
Read Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability by Kingsuk Maitra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04902.txt.
Description
Kingsuk Maitra, writing alone, proposes a physics-inspired way to read transformers in this 2026 paper. Mechanistic interpretability tries to map exactly how a neural network computes. Maitra treats the transformer as an electrical circuit with changing signals, and adds a momentum term that tracks how each token's representation moves from one step to the next.
The added term acts like a high-pass filter, letting through fast changes while blocking slow ones. Maitra proves the fast and slow signals stay in separate frequency bands, which lets a single layer do pattern matching that normally needs two. In more than 5,100 experiments, a small model with momentum nearly matched a model almost three times its size. The paper frames the method as a bridge between generative AI, classical physics, and signal processing.
Catalog record
| CREATOR | Kingsuk Maitra |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04902