Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
Anamika Paul Rupa · 2026
Read Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold by Anamika Paul Rupa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00230.txt.
Introduction
Rupa identifies a stable predictor for neural collapse: collapse sets in when mean feature norm reaches a critical value specific to each model and dataset. Crossing below that value precedes collapse onset by a mean 62 epochs, and an intervention test confirms it as a stable attractor. Depth affects collapse speed unevenly, activation shapes speed and the critical value, weight decay splits into three regimes, and width speeds collapse while shifting the value by at most 13 percent. ResNet-20 on MNIST shows the strongest architecture effect, 458 percent against 68 percent on CIFAR-10.
Catalog record
| CREATOR | Anamika Paul Rupa |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49337 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00230
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholds Thanks: This manuscript is under review at IEEE Access. Anamika Paul Rupa Abstract Neural collapse (NC)—the convergence of penultimate-layer features to a simplex equiangular tight frame—is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model–dataset-specific critical value, fn ∗ \mathrm{fn}^{*} , that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8 % <8\% ); training dynamics primarily affect the rate at which fn \mathrm{fn} approaches fn ∗ \mathrm{fn}^{*} , rather than the value itself. In standard training trajectories, the crossing of fn \mathrm{fn} below fn ∗ \mathrm{fn}^{*} consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn ∗ \mathrm{fn}^{*} is a stable attractor of the gradient flow—perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction ( p > 0.2 p{>}0.2 ). Completing the (architecture) × \times (dataset) grid reveals the paper’s strongest result: ResNet-20 on MNIST gives fn ∗ \mathrm{fn}^{*} = 5.867 =5.867 —a + 458 % +458\% architecture effect versus only + 68 % +68\% on CIFAR-10. The grid is strongly non-additive; fn ∗ \mathrm{fn}^{*} cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn ∗ \mathrm{fn}^{*} ; (3) weight decay defines a three-regime phase diagram—too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn ∗ \mathrm{fn}^{*} by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks. Keywords: neural collapse, feature norm, grokking, deep learning dynamics, representation learning, weight decay, training dynamics, MLP 1 Introduction Neural collapse (NC), first documented by Papyan et al. [ 15 ] , is a striking geometric regularity that emerges at the end of deep network training: penultimate-layer features converge to a simplex equiangular tight frame (ETF), within-class variability vanishes (NC1), and classifier weights align with class means (NC3). The endpoint of NC is now well characterised theoretically [ 25 , 4 , 24 , 8 ] , yet a fundamental question has received almost no systematic attention: what controls when and how fast NC emerges as a function of architecture and training hyperparameters? This question has direct practical consequences. If NC timing is unpredictable, practitioners cannot anticipate how changes to depth, width, activation, or regularisation will affect the representational reorganisation their networks undergo. If, conversely, NC timing is governed by a simple, measurable quantity, that quantity could serve as an actionable diagnostic. Our approach is motivated by an analogy with grokking [ 16 , 11 ] , a superficially different delayed-generalisation phenomenon in which networks first memorise then abruptly generalise. Manir and Rupa [ 11 ] showed that grokking timing is predicted by the RMS weight norm crossing a model-specific threshold, with weight decay controlling the approach rate but not the threshold value. We test the direct analogue for NC: does a feature norm threshold play the same predictive role for collapse onset? We investigate through five controlled experiments: 1. Does depth affect NC speed monotonically? 2. Does activation function shape the collapsed geometry, not just the speed? 3. Is there a Goldilocks weight decay zone for NC? 4. Is NC timing predicted by a feature norm threshold, as in grokking? 5. What primarily determines \mathrm{fn}^{*} —width, architecture family, or dataset? Our central claim is precise: neural collapse occurs when the mean feature norm reaches a model–dataset-specific critical value that is largely invariant to training dynamics but predictive of collapse timing. Training conditions (depth, activation, weight decay, width) control T_{\mathrm{NC}} , but \mathrm{fn}^{*} concentrates within each (model, dataset) pair (CV <8\% ) and \mathrm{fn} crossing below \mathrm{fn}^{*} predicts T_{\mathrm{NC}} with a mean lead of 62 epochs. The between-pair gap is highly significant ( F{=}55.88 , p{<}0.0001 ). The most striking result comes from completing the (architecture) \times (dataset) grid: ResNet-20 on MNIST gives \mathrm{fn}^{*} =5.867 , a +458% architecture effect compared to MLP-5 on the same dataset—yet the same architecture on CIFAR-10 gives only +68\% . This 6.7\times difference in effect size across datasets rules out a simple additive decomposition of \mathrm{fn}^{*} into architecture and dataset contributions. The five structural regularities match the grokking weight norm threshold precisely, suggesting a unified account of delayed representational reorganisation. The remainder of the paper is organised as follows. Section II defines NC metrics and the two-phase training protocol. Section III reviews related work. Section IV describes the experimental setup. Section V presents results across five experiments. Section VI discusses the grokking parallel, threshold interpretation, and limitations. Section VII concludes. 2 Background 2.1 NC Metrics Let \mathbf{h}_{i}\in\mathbb{R}^{d} be the penultimate-layer feature for training example i , \boldsymbol{\mu}_{c} the class-conditional mean, \boldsymbol{\mu}_{G} the global mean, and K the number of classes. NC1 : \text{Tr}(\Sigma_{W})/\text{Tr}(\Sigma_{B})\to 0 , where \Sigma_{W} , \Sigma_{B} are within- and between-class covariance matrices. NC2 : \frac{1}{K(K-1)}\sum_{i\neq j}|\cos(\mathbf{m}_{i},\mathbf{m}_{j})+1/(K{-}1)| \to 0 , where \mathbf{m}_{c}=\boldsymbol{\mu}_{c}-\boldsymbol{\mu}_{G} . NC3 : 1-\frac{1}{K}\sum_{c}\cos(\hat{\mathbf{w}}_{c},\hat{\mathbf{m}}_{c})\to 0 . NC1 is our primary metric; NC2 and NC3 are reported for the baseline to confirm all three properties emerge together. We additionally track mean feature norm \mathrm{fn}=\frac{1}{N}\sum_{i}\|\mathbf{h}_{i}\|_{2} as a dynamical predictor of collapse onset. Definition (Feature Norm Threshold). We define \mathrm{fn}^{*}=\mathrm{fn} at epoch T_{\mathrm{NC}} —the first epoch with NC1 <\varepsilon — and ask whether \mathrm{fn}^{*} is approximately constant across training conditions within a fixed (model, dataset) pair. We use “threshold” throughout in an operational, predictive sense: \mathrm{fn}^{*} is the \mathrm{fn} value at which collapse is imminent, not a causal trigger. The intervention experiment (Section VI-C) confirms that \mathrm{fn}^{*} is a gradient-flow attractor—both \mathrm{fn} trajectories above and below \mathrm{fn}^{*} converge to it—so the threshold label denotes a predictive landmark, not a mechanism. \mathrm{fn} approaching \mathrm{fn}^{*} is associated with impending collapse, but we make no causal claim (see Section VI-C). 2.2 NC1 Thresholds and Threshold Robustness We use NC1 <0.01 as the collapse criterion for ReLU networks and NC1 <0.05 for GELU, Tanh, and shallow networks (depth \leq 3 ). This distinction reflects an empirical ceiling: non-ReLU activations and shallow MLPs plateau in the NC1 range 0.03–0.08 rather than below 0.01, consistent with weaker global collapse under these conditions [ 9 ] . A natural concern is whether reported differences in \mathrm{fn}^{*} across conditions are artefacts of using different NC1 thresholds rather than genuine differences in collapse geometry. We address this with two arguments (Table 1 ). Threshold robustness for ReLU configs. For all ReLU networks—where NC1 reaches 0.01 cleanly—we verify that \mathrm{fn}^{*} is stable across the range NC1 \in\{0.01,0.02\} . Across six condition groups (depths 5 and 7, widths 128 and 256, weight decays 10^{-5} and 5\times 10^{-5} ), the ratio \mathrm{fn}^{*} (NC1 <0.02 ) / \mathrm{fn}^{*} (NC1 <0.01 ) ranges from 0.86 to 1.27 with a mean of 1.10. This \pm 10\% variation is smaller than the between-pair gap (44%)