Grokking and Generalization Collapse: Insights from HTSR theory Hari K. Prakash, Charles H. Martin Abstract We study the well-known grokking phenomena in neural networks (NNs) using a 3-layer MLP trained on 1 k-sample subset of MNIST, with and without weight decay, and discover a novel third phase — anti-grokking –that occurs very late in training and resembles but is distinct from the familiar pre-grokking phases: test accuracy collapses while training accuracy stays perfect. This late-stage collapse is distinct, however, from the known pre-grokking and grokking phases, and is not detected by other proposed grokking progress measures. Leveraging Heavy-Tailed Self-Regularization (HTSR) through the open-source WeightWatcher tool, we show that the HTSR layer quality metric α \alpha alone delineates all three phases, whereas the best competing metrics detect only the first two. The anti-grokking is revealed by training for 10 7 10^{7} and is invariably heralded by α < 2 \alpha<2 and the appearance of Correlation Traps —outlier singular values in the randomized layer weight matrices that make the layer weight matrix atypical and signal overfitting of the training set. Such traps are verified by visual inspection of the layer-wise empirical spectral densities, and using Kolmogorov–Smirnov tests on randomized spectra. Comparative metrics, including activation sparsity, absolute weight entropy, circuit complexity, and l 2 l^{2} weight norms track pre-grokking and grokking but fail to distinguish grokking from anti-grokking. This discovery provides a way to measure overfitting and generalization collapse without direct access to the test data. These results strengthen the claim that the HTSR α \alpha provides universal layer-convergence target at α ≈ 2 \alpha\!\approx\!2 and underscore the value of using the HTSR alpha ( α ) (\alpha) metric as a measure of generalization. 1 Introduction Grokking is an intriguing phenomenon where a neural network achieves near-perfect training accuracy quickly, yet the test accuracy lags significantly, often near chance level, before abruptly surging towards high generalization power2022grokking . Figure 1 illustrates this for a depth-3, width-200 ReLU MLP trained on a subset of MNIST. To dissect this phenomenon and uncover deeper dynamics, our primary analytical lens is the recently developed theory of Heavy-Tailed Self-Regularization ( HTSR ), following Martin et al. martin2021implicit . The HTSR theory examines the empirical spectral density (ESD) of individual layer weight matrices (\mathbf{W}) , quantified by the heavy-tailed power law (PL) exponent \alpha . We find \alpha provides a sensitive measure of correlation structure within layers, tracking the transition into the grokking phase, and crucially, predicting a subsequent decrease in generalization. For comparative context, we also investigate several other methodologies: 1. Weight Norm Analysis: Motivated by studies like Liu et al. liu2023omnigrokgrokkingalgorithmicdata , we examine the l^{2} norm of the weights. We observe that grokking occurs even without weight decay (leading to an increasing norm), suggesting weight norm alone is not a complete explanation, confirming the weight-norm related findings by Golechha golechha2024progressmeasuresgrokkingrealworld . 2. Progress Measures: We utilize metrics proposed by Golechha golechha2024progressmeasuresgrokkingrealworld .—Activation Sparsity, Absolute Weight Entropy, and Approximate Local Circuit Complexity—which capture broader structural and functional changes in the network during training. Figure 1 : The three phases of grokking. Training curves for a depth-3, width-200 MLP on MNIST. The initial pre-grokking phase (grey): training accuracy (red line) surges at 10^{2} steps, saturating between 10^{4}-10^{5} steps, while test accuracy (purple line) remains low; the grokking phase (yellow): with test accuracy rapidly increasing after \sim 10^{5} steps, and reaching a maximum at 10^{6} steps; and the newly revealed late-stage anti-grokking phase (green): test accuracy collapses (to 0.5 ). Our Contributions: Our work makes several related contributions that helps explain the underlying mechanisms associated with the grokking phenomena: 1. By extending training significantly (up to 10^{7} steps) under zero weight decay WD=0 , we identify and characterize late-stage generalization collapse : a substantial drop in test accuracy long after initial grokking, despite perfect training accuracy and a continually increasing l^{2} weight norm. We call this anti-grokking. 2. We show that the HTSR layer quality metric \alpha (the heavy-tailed power-law (PL) exponent), effectively tracks the grokking transition under both the traditional setting of weight decay WD>0 and zero weight decay WD=0 , outperforming the l^{2} weight norm and the other progress metrics. Only the HTSR \alpha can distinguish between all 3 phases of grokking. 3. We identify the mechanism of the pre-grokking phase, where the training accuracy is perfect but the model does not generalize. This phase occurs because only a subset of the model layers are well trained (i.e. \alpha\leq 4 ), whereas at least one layer is underfit (i.e. \alpha\geq 5 ). Moreover, the layers can show great variability between training runs, indicating their instability. Importantly, the layer \alpha ’s here are distinct from those in the anti-grokking phases, despite both phases having perfect training accuracy and low test accuracy. . 4. We demonstrate that when the HTSR PL exponent \alpha<2 , this identifies the collapse. Also, in this phase, we observe the presence of anomalous rank-one (or greater) perturbations in one or more underlying layer weight matrices \mathbf{W} . We call these correlation traps and identify them by randomizing \mathbf{W} elementwise, forming \mathbf{W}^{rand} , and looking for unusually large eigenvalues, \lambda_{trap}\gg\lambda^{+} (where \lambda^{+} is the right-most edge of the associated Marchenko-Pastur (MP) distribution martin2025setol ). 2 Related Work Grokking power2022grokking , the delayed emergence of generalization well after training accuracy saturation, has prompted significant research into its underlying mechanisms. Initial studies often explored grokking in algorithmic tasks merrill2023tale ; nanda2023progress ; varma2023explaining , frequently linking the phenomenon to the presence of weight decay (WD) which favors simpler, lower-norm solutions liu2023omnigrokgrokkingalgorithmicdata . Other approaches include mechanistic interpretability nanda2023progress and analyses identifying competing memorization and generalization circuits varma2023explaining ; merrill2023tale . Varma et al. varma2023explaining defined ’ungrokking’ as generalization loss when retraining a grokked network on a smaller dataset ( D0 control experiment, as detailed in Appendix C , a similar inflection in A_{s} occurs where test accuracy, after a slight initial decrease from its peak, subsequently plateaus rather than undergoing a catastrophic collapse as seen in the WD=0 case. Therefore, observing this dip in A_{s} alone does not allow one to distinguish whether test accuracy will catastrophically decline or stabilize, suggesting it primarily indicates that some form of transitional change has occurred around the point of maximum generalization, rather than predicting the specific nature of the subsequent trajectory. Our findings indicate limitations in the other two comparitive metrics for tracking the anti-grokking phase. Absolute Weight Entropy ( H_{\text{abs}}(\mathbf{W}) ), despite its suggested link to generalization ( golechha2024progressmeasuresgrokkingrealworld ) , also decreases sharply during the collapse, thus not reliably distinguishing this anti-grokking phase. Similarly, \Lambda_{\text{LC}} ( golechha2024progressmeasuresgrokkingrealworld ) remains low throughout the collapse, failing to reflect the performance degradation. We also confirm, consistent with ( golechha2024progressmeasuresgrokkingrealworld ) , that grokking occurs robustly even with increasing weight norms and no weight decay. 4.2 Correlation Traps and Anti-Grokking To better understand the origin of anti-grokking (generalization collapse), it is instructive to look for evidence of potential overfitting in the layer weight matrices \mathbf{W} , in the form correlation traps. As described in Section 3.1 , we analyze the eigenvalues \{\lambda_{i}\} of the randomized weight matrices \mathbf{W}^{rand} derived from each layer’s weight matrix \mathbf{W} for layers FC1 and FC2. Table 2 : Average number of detected correlation traps in layers FC1 and FC2 at the right edge of of the three grokking phases: Pre-grokking \sim 10^{5} steps, Grokking 10^{6} steps, and Anti-grokking 10^{7} steps. Results shown for both experiments, with (WD>0) and without WD=0 weight decay. Model, Layer Pre-grokking Grokking (Max Test Acc.) Anti-grokking (Collapse) WD=0 , FC1 0\pm 0 1\pm 0 7.5\pm 5.6 WD=0 , FC2 0\pm 0 1\pm 0 1\pm 0 WD>0 , FC1 0 0 2.0\pm 0.0 WD>0 , FC2 0 0 1.0\pm 0.0 As show in Table 2 , for both layers, FC1 and FC2, and for both experiments, with and without weight decay, neither layer shows evidence of correlation traps until the anti-grokking phase. The presence of such traps corresponds to HTSR \alpha<2 for these layers, as predicted by previous work martin2025setol . Further statistical analysis for the FC2 layers is provided in Appendix D . The presence of correlation traps, combined with \alpha<2 , is a definitive signal indicating the model is in the anti-grokking phase. 5 Conclusion This study investigated the well-known grokking phenomena in neural networks (NN) under the lens of the recently developed theory of Heavy-Tailed Self Regularization ( HTSR ) martin2021implicit . Previous work has attempted to explain grokking (using the l^{2} norm), but only succeeds in the presence of weight decay ( WD ), and has been unable to explain grokking without weight decay liu2023omnigrokgrokkingalgorithmicdata ; golechha2024progressmeasuresgrokkingrealworld . For this reason, we have studied the long-term generalization dynamics of the grokking phenomena both with weight decay ( WD>0 ) and without ( WD=0 ). We compare the application of the HTSR theory to the l^{2} norm and several previous proposed metrics. liu2023omnigrokgrokkingalgorithmicdata Our primary finding is that the HTSR layer quality metric \alpha can effectively track grokking both with and without weight decay. In particular, the HTSR \alpha tracks the initial grokking transition and subsequent performance dips in both cases ( WD=0\;,WD>0 ) and, in doing so, offers new insights into the grokking phenomena. Moreover, and critically, in the WD=0 setting, the HTSR \alpha also provides an early indication of a novel late-stage generalization collapse , called anti-grokking . This collapse is characterized by a significant drop in test accuracy despite sustained perfect training accuracy (and a large l^{2} norm), and is observed after extensive training (up to 10^{7} steps). We also examined several other grokking progress measures, in addition to the l^{2} norm liu2023omnigrokgrokkingalgorithmicdata , including Activation Sparsity A_{s} , Absolute Weight Entropy H_{abs}(W) , and Approximate Local Circuit Complexity \Lambda_{LC} golechha2024progressmeasuresgrokkingrealworld . Although A_{s} and \Lambda_{LC} captured initial training and grokking phases, and do change at the anti-grokking transition, they failed to unambiguously predict the appearance and/or presence of anti-grokking. In examining the HTSR results on all 3 phases of grokking, we propose a new explanation of the grokking phenomena. During the first phase, pre-grokking, where only training accuracy saturates, only a subset of the individual layers will converge, and only far enough (i.e, \alpha\approx 4 ) to describe the training data, while other layers will appear almost random (i.e, \alpha\approx 5 ). Importantly, some layers will be more important for generalization than others, and these will not have converged very well at all. During the grokking phase, when the test accuracy is maximal, all important layers will converge extremely well, with \alpha metrics approach the optimal value with \alpha\approx 2.0 –exactly as predicted by the HTSR theory. In the third anti-grokking phase, where the test accuracy drops substantially, one or more layer will overfit the data in some yet undetermined way). They will have \alpha<2 , and may exhibit correlation traps (and/or even rank collapse). (Note these results are also supported by recent theoretical developments in HTSR (and SETOL) theory martin2025setol .) In particular, we consider the implications of observing numerous correlation traps in the anti-grokking phase. The ’traps’ are anomalous rank-one (or greater) perturbations in the weight matrix \mathbf{W} , causing a large mean-shift in underlying distribution of elements: \mathbb{E}[W_{ij}]\rightarrow\textit{large} and, ’pushing’ the ESD into the VHT phase where \alpha<2 . The large shift in \mathbb{E}[W_{ij}]\rightarrow\textit{large} indicates that the distribution of weights is atypical . That is, different random samples of the weights could have very different means. And as with any statistical estimator, an atypical distribution will not generalize well. (Similar results have been seen in training a similar model with very large learning rates martin2025setol .) Consequently, it is hypothesized that layers with large numbers of correlation traps are overfit to the training data (in some unspecified way), and hurt the overall model test accuracy. These results underscore the utility of HTSR for monitoring and understanding long-term generalization stability across different regularization schemes, with a particular strength in identifying potential catastrophic collapse. The observed layer-specific changes in \alpha during the WD=0 collapse suggest that potential over-fitting may develop deep into training. While our current findings are based on a specific MLP architecture and MNIST subset, further research should validate these observations across diverse datasets, architectures, hyperparameter configurations, and optimizers. Promising future work includes developing \alpha -guided adaptive training strategies. Additionally, designing differentiable regularizers or loss terms based on \alpha could potentially enable faster and more stable generalization, for instance, by encouraging convergence towards \alpha\approx 2 . Appendix A Experimental Setup We train a Multi-Layer Perceptron (MLP) on a subset of the MNIST dataset using the hyperparameters detailed in Table 3 . The training subset is constructed by randomly selecting 100 samples from each of the 10 MNIST classes, ensuring a balanced dataset of 1,000 unique training points. This was run on an Nvidia Quadro P2000 and took approximately 11 hours. A considerable part of the time is due to the speed of saving the measures. Table 3 : Experimental hyperparameters used in the study (details in Appendix A ). Parameter Value Network Architecture Fully Connected MLP Depth 3 Linear layers (Input \to Hidden1 \to Hidden2 \to Output) Width 200 hidden units per hidden layer Activation Function ReLU (Rectified Linear Unit) Input Layer Size 784 (Flattened MNIST image 28\times 28 ) Output Layer Size 10 (MNIST digits 0-9) Weight Initialization Default PyTorch (Kaiming Uniform for weights), parameters scaled by 8.0 Bias Initialization Default PyTorch (Uniform), then scaled by 8.0 Dataset MNIST Training Points 1,000 (100 per class, stratified random sampling) Test Points Standard MNIST test set (10,000 samples) Batch Size 200 Loss Function Mean Squared Error (MSE) with one-hot encoded targets Optimizer AdamW Learning Rate (LR) 5\times 10^{-4} Weight Decay (WD) 0.0 (for main results), 0.01 (for Appendix C comparison) AdamW \beta_{1} 0.9 (PyTorch default) AdamW \beta_{2} 0.999 (PyTorch default) AdamW \epsilon 10^{-8} (PyTorch default) Optimization Steps 10^{7} Data Type (PyTorch) ‘torch.float64‘ Random Seed 0 (for all libraries) Software Framework PyTorch HTSR Tool WeightWatcher v0.7.5.5 [ 8 ] Note on Weight Decay: The primary results presented in this paper, particularly those demonstrating grokking followed by late-stage generalization collapse (Figure 1 ), were obtained with weight decay explicitly set to 0. This allows observation of the learning dynamics driven purely by the optimizer and the loss landscape while exhibiting both phenomena, whereas the other proposed measures fail to detect the grokking transition of increasing test accuracy. Runs with non-zero weight decay (e.g., WD=0.01 , see Appendix C ) were also performed for comparison, showing different dynamics but confirming the general utility of HTSR . Appendix B Comparative Grokking Progress Metrics and Measures Weight Norm Analysis Following observations that weight decay can influence grokking [ 6 ] , we monitor the l^{2} norm of the network’s weights, ||\mathbf{W}||_{2}=\sqrt{\sum_{l}||\mathbf{W}_{l}||_{F}^{2}}, (7) throughout training. We specifically run experiments with weight decay disabled ( WD=0 ) to isolate the effect of the optimization dynamics on the norm itself. Activation Sparsity. For a given layer with activations b_{i,j} (representing the activation of neuron j for input example i ), the activation sparsity A_{s} is defined as: A_{s}=\frac{1}{T}\sum_{i=1}^{T}\frac{1}{n}\sum_{j=1}^{n}\mathbf{1}(b_{i,j}<\tau), (8) where T is the number of training examples, n is the number of neurons in the layer, \tau is a chosen threshold, and \mathbf{1}(\cdot) is the indicator function. This metric measures neuron inactivity. Prior studies have linked activation sparsity to generalization [ 5 , 12 , 14 ] and reported specific dynamics such as plateauing before grokking [ 2 ] or an increase preceding a rise in test loss [ 3 ] . Absolute Weight Entropy. For a weight matrix W\in\mathbb{R}^{m\times n} , the absolute weight entropy H_{abs}(W) is given by: H_{abs}(W)=-\sum_{i=1}^{m}\sum_{j=1}^{n}|w_{i,j}|\log|w_{i,j}|. (9) This entropy quantifies the spread of absolute weight magnitudes. Golechha [ 2 ] suggested its sharp decrease signals generalization. Approximate Local Circuit Complexity. Let L^{(W)}(x) denote the output logits for input x using weights W , and let L^{(W^{\prime})}(x) denote the logits when 10% of the weights are set to zero (forming W^{\prime} ). The approximate local circuit complexity, denoted \Lambda_{LC} , is the summed KL divergence: \Lambda_{LC}=\sum_{k=1}^{N_{data}}\sum_{j\in\mathcal{C}}\text{Pr}\bigl(j|L^{(W)}(x_{k})\bigr)\log\frac{\text{Pr}\bigl(j|L^{(W)}(x_{k})\bigr)}{\text{Pr}\bigl(j|L^{(W^{\prime})}(x_{k})\bigr)}. (10) Here, N_{data} is the number of training examples x_{k} , \mathcal{C} is the set of classes, and \text{Pr}(j|L(x)) is the probability of class j derived from the logits L(x) (e.g., via softmax). This measure captures output sensitivity to minor weight perturbations. Lower \Lambda_{LC} has been linked to stable, generalizable representations [ 2 ] . Appendix C Experiment with Weight Decay To further understand the influence of weight decay on the observed generalization dynamics and the behavior of our tracked metrics, we conducted an experiment identical to our main study ( WD=0 ) but with a small amount of weight decay ( WD=0.01 ) applied. The training curves and metric evolutions for this WD=0.01 experiment are presented in Figures 6 , and 7 . A key characteristic of training with weight decay is the tendency for the l^{2} norm of the weights to decrease over time, or stabilize at a lower value, which is observed in this experiment (Figure 7 ). This contrasts with the continuously increasing l^{2} weight norm seen in our primary WD=0 experiments. In this WD=0.01 regime, the network still achieves a high level of test accuracy. Notably, after the initial grokking phase, the test accuracy slightly decreases and then enters a prolonged plateau, maintaining near peak performance for a significant number of optimization steps (Figure 6 ). Correspondingly, the average heavy-tail exponent, \alpha , also exhibits the decrease and a distinct plateau around the critical value of \alpha\approx 2 during this period (Figure 6 , top left panel). The other progress measures considered—Activation Sparsity and Approximate Local Circuit Complexity—also tend to plateau or stabilize during this phase of peak test performance in the WD=0.01 setting (Figure 7 ). This contrasts with the WD=0 scenario where, despite eventual grokking, the system does not find such a stable long-term plateau and instead proceeds towards a late-stage generalization collapse. The observation that \alpha (and other metrics) plateau in conjunction with peak, stable test accuracy under traditional weight decay settings aligns with some existing understanding of well-regularized training. While HTSR and the \alpha exponent provide valuable insights in both regimes, its unique capability to signal impending collapse in the absence of weight decay underscores its importance for understanding layer dynamics under various scenarios. Figure 6 : HTSR \alpha exponent evolution for the MLP trained with WD=0.01. Figure 7 : Progress measures (Activation Sparsity, Weight Entropy, Circuit Complexity) and l^{2} Weight Norm for the MLP trained with WD=0.01. Appendix D Statistical Analysis and Validation of Correlation Traps Here, to further validate the presence of correlation traps for the zero weight decay WD=0 experiment , we report the results of statistical tests designed to determine if the randomized ESD of the \mathbf{W}^{rand} fits an MP distribution or not. Briefly we fit the ESD to a MP distribution and report the fitted variance \sigma_{mp} , the Kolmogorov-Smirnov (KS) statistic of the fit, and the p-value for the MP fit as the null model. We also report the number of correlation traps, as determined using the open-source WeightWatcher tool [ 8 ] . Results for layer FC1 are presented in Table 4 . Results for FC2 are similar (not shown). Additional details are provided in the supplementary material. Table 4 : Statistical validation of correlation traps. Selected results for layer FC1 at different training stages for zero weight decay ( WD=0 ) experiment. MP Variance (\sigma_{MP}) Kolmogorov-Smirnov (KS) test statistic, p-value for MP fit, and number of detected correlation traps. Pre-grokking \sim 10^{5} steps, Grokking 10^{6} steps, and Anti-grokking 10^{7} steps, Model State MP variance (\sigma_{mp}) KS Statistic p-value # Traps Pre-Grokking \approx 1.002 0.0120 \approx 1.0 0 Grokking (Max Test Accuracy) \approx 0.999 0.0212 \approx 1.0 0 Anti-Grokking (Collapse) \approx 0.949 0.3044 1.877\times 10^{-5} 9 Initial Layer State (Pre-Grokking WD=0 ): Immediately after initialization, the network weights are expected to be largely random, and their ESD should conform well to the MP distribution. Figure 2 (Right) shows an MP fit to an ESD from a representative layer \mathbf{W}^{rand} of the newly initialized model. A KS test comparing this empirical ESD to the fitted MP distribution (using \sigma_{mp}\approx 1.0024 as estimated by WeightWatcher ) yielded a KS statistic of 0.0120 and a p-value \approx 1.0 . This high p-value indicates this ESD is statistically consistent with the MP distribution, as expected. Best Layer State (Grokking phase WD=0 ): As the network learns and reaches its maximum test accuracy, significant structure develops in the elements of the weight matrices W_{i,j} . This can be seen by randomizing the layer weight matrix elementwise, \mathbf{W}\rightarrow\mathbf{W}^{rand} , and plotting ESD, and looking for deviations from the theoretical MP distribution. The ESD now typically exhibits a pronounced heavy tail, with eigenvalues extending beyond the bulk region that might be approximated by an MP fit. For our model at peak test accuracy, the KS test against a fitted MP model ( \sigma_{mp}\approx 0.999 ) resulted in a KS statistic of 0.0212 and a p-value \approx 1 . Again, this is an MP distribution. Final Layer State (Anti-Grokking phase WD=0 ): In the late-stage of training, as the model undergoes generalization collapse and enters an over-correlated state (characterized by \alpha<2 ), the ESD of \mathbf{W}^{rand} structure continues to reflect a non-random configuration. The KS test for the final model against an MP fit (with an estimated \sigma_{mp}\approx 2 ) yielded a KS statistic of 0.3044 and a p-value of 1.877\times 10^{-5} Figure 3 (Right) . This result further confirms that the network’s structure remains significantly different from a random matrix baseline, consistent with the highly correlated or near rank-collapsed state indicated by our HTSR analysis. These quantitative comparisons demonstrate a transition from an initially random-like state (consistent with MPD) to progressively more structured, non-random states as learning occurs and eventually leads to over-correlation. The inability of the MP distribution to describe these learned features, especially the heavy tails, necessitates the use of tools like the HTSR theory, the PL exponent \alpha , and the open-source WeightWatcher tool, to properly characterize these complex correlation structures and their relationship to generalization performance. Appendix E Limitations Our study, while providing insights into generalization dynamics via Heavy-Tailed Self-Regularization ( HTSR ), has limitations that define important avenues for future research. The empirical findings are primarily derived from a specific three-layer MLP architecture trained on an MNIST subset. Consequently, the generalizability of the observed \alpha trajectories and their specific predictive power for phenomena like grokking and late-stage generalization collapse warrants further validation across a wider range of model architectures (e.g., CNNs, Transformers), datasets, tasks, and diverse training configurations, including different optimizers and hyperparameter settings. Furthermore, HTSR is an empirically-grounded, phenomenological framework, supported theoretically with a novel application of Random Matrix Theory (RMT). While its correlations between the heavy-tailed PL exponent \alpha and network generalization states are compelling, the interpretation requires careful consideration of context. For instance, while well-generalized models often exhibit \alpha values within the range (e.g., 2\leq\alpha\leq 6 ), and \alpha\approx 2 is frequently associated with optimal performance or critical transitions, this is not a strictly bidirectional implication. It is conceivable that layers or models might exhibit \alpha values near or even below 2 (typically indicating over-correlation) yet display suboptimal generalization. Other very-well trained models may have layers fairly large alphas. This is not yet fully understood. This highlights that while \alpha provides strong correlational insights into learning phases and stability, the precise mapping of specific \alpha values to absolute performance levels can be context-dependent and is an area for ongoing refinement of the theory (see [ 9 ] ). Our work contributes observations within specific phenomena, acknowledging that the broader applicability and predictive nuances of the HTSR theory will benefit from continued exploration. These limitations underscore the importance of ongoing empirical and theoretical work to further refine, validate, and extend the understanding of HTSR theory in deep learning.