Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking
Yongzhong Xu · 2026
Read Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking by Yongzhong Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08119.txt.
Description
Yongzhong Xu tests a recent theory of grokking, the delayed jump in generalization seen when small networks train far past memorization. Tian's repulsion theorem predicts that during feature learning, similar features push each other apart through negative entries in a certain inverse matrix. The theorem does not say when the effect becomes visible or whether updates carry a spectral trace.
On a modular addition task, Xu finds a clear split. The predicted sign rule held strongly on the most similar feature pairs, with match rates rising toward one across seeds and activations. The spectral signature in parameter updates, though, depended heavily on the activation function used.
Catalog record
| CREATOR | Yongzhong Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08119