Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

Yongzhong Xu · 2026

Read Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking by Yongzhong Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08119.txt.

Description

Yongzhong Xu tests a recent theory of grokking, the delayed jump in generalization seen when small networks train far past memorization. Tian's repulsion theorem predicts that during feature learning, similar features push each other apart through negative entries in a certain inverse matrix. The theorem does not say when the effect becomes visible or whether updates carry a spectral trace.

On a modular addition task, Xu finds a clear split. The predicted sign rule held strongly on the most similar feature pairs, with match rates rising toward one across seeds and activations. The spectral signature in parameter updates, though, depended heavily on the activation function used.

Catalog record

CREATORYongzhong Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08119

Browse the full library · About the Tower of Babel