Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability

Kingsuk Maitra · 2026

Read Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability by Kingsuk Maitra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04902.txt.

Description

Kingsuk Maitra, writing alone, proposes a physics-inspired way to read transformers in this 2026 paper. Mechanistic interpretability tries to map exactly how a neural network computes. Maitra treats the transformer as an electrical circuit with changing signals, and adds a momentum term that tracks how each token's representation moves from one step to the next.

The added term acts like a high-pass filter, letting through fast changes while blocking slow ones. Maitra proves the fast and slow signals stay in separate frequency bands, which lets a single layer do pattern matching that normally needs two. In more than 5,100 experiments, a small model with momentum nearly matched a model almost three times its size. The paper frames the method as a bridge between generative AI, classical physics, and signal processing.

Catalog record

CREATORKingsuk Maitra
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04902

Browse the full library · About the Tower of Babel