A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio
Ali Shehper, Ashish Vaswani · 2026
Read A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio by Ali Shehper, Ashish Vaswani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28975.txt.
Introduction
Ali Shehper and Ashish Vaswani study the log-alignment ratio, a measure of how model weights line up with activations. Recast as overlap between a weight spectrum and an activation spectrum, it tracks the shift between memorization and generalization in two settings. In grokking it predicts the effective dimension of the learned function. In 3B-parameter language-model pretraining, its drift from a non-overfitting baseline tracks the generalization gap. The ratio needs only forward-pass numbers, adds tiny overhead, and needs no held-out test data.
Catalog record
| CREATOR | Ali Shehper, Ashish Vaswani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59941 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio Ali Shehper, Ashish Vaswani Abstract We study the log-alignment ratio (LAR), a measure of parameter–activation alignment, introduced in parameterization theory. We reformulate it as the overlap between a weight spectrum p p of the normalized squared singular values of a matrix and an activation spectrum q q of the normalized squared projections of inputs onto its singular directions. We show that unembedding LAR tracks the transition between memorization and generalization in two different settings by capturing the spread of p p and q q during training. In grokking, LAR predicts the effective dimension of the learned function: k ≈ n 2 ( 1 − LAR ) k\approx n^{2(1-\mathrm{LAR})} , where n n is the input dimension of the matrix. In 3B-parameter language model pre-training, its deviation from a non-overfitting baseline tracks the generalization gap, and its rate of decline increases as overfitting approaches. LAR is computable from quantities available during the forward pass with negligible computational overhead, and requires no held-out validation data. 1 Introduction Large-scale pre-training has driven the most visible recent advances in neural network capability ( 12 ; 9 ) , and as model and dataset sizes continue to grow, the cost of every training run grows with them. This has made the study of training dynamics, and the design of cheap diagnostic tools for monitoring them, unusually valuable: a metric that can flag problems such as overfitting, instability, or saturation from training-time quantities alone, without requiring expensive held-out evaluation, can save compute and time for practitioners. A complementary line of work on neural network parameterizations , which prescribe how initialization, parameter multipliers, and learning rates should scale with width and depth, offers a principled approach to transferring hyperparameters from small to large models. 28 proposed the maximal update parametrization ( \mu P ), which enables hyperparameter transfer under the assumption that updates at the readout layer are fully aligned with the incoming activations. 6 showed that this alignment assumption is stronger than necessary and, by relaxing it, obtained a standard-parameterization recipe that consistently outperformed \mu P across tens of thousands of experiments at scales up to 26.8B parameters. As part of this analysis they introduced the log-alignment ratio (LAR) as an empirical measure of how strongly a layer’s parameter matrix aligns with its input activations during training. While 6 used LAR as a proxy for parameter-activation alignment, we discover in this paper that the metric carries a second, previously-unexplored meaning: namely, it measures how diffuse or concentrated the layer’s activations and weight spectra are. More precisely, we show that LAR measures the overlap between a weight spectrum p and an activation spectrum q : it is large when both spectra are concentrated on the same small set of directions, and small when either is diffuse or when the two are misaligned. This insight makes LAR a natural candidate to measure generalization — the extent to which a model generalizes beyond its training data rather than overfitting to it. Empirically, we observe that well-generalizing networks tend to concentrate their computation on a small number of directions, while overfitting networks tend to spread it across many. Most previously-studied metrics for this behavior cover either the weight spectrum or the activation spectrum, but not both, and are computationally expensive as they require full SVD of the weight matrix or a PCA of the activations. In contrast, LAR looks at both weights and activations and is computable from quantities already available in the model’s forward pass, with negligible computational overhead. We study the relationship between generalization and the LAR of a model’s unembedding matrix in two different settings. First, on small algorithmic tasks exhibiting grokking ( 22 ) , we find that optimization concentrates p and q on a shared set of directions, which causes LAR to grow. As the number k of directions becomes small, grokking may occur, depending on factors such as task difficulty and training dataset size. LAR predicts the dimensionality of the learned function, k\approx n^{2(1-\mathrm{LAR})} , which we compare against k_{95} — the number of principal components explaining 95% of the unembedding input activation variance. We find close agreement for final checkpoints across all experiments, including 12 grokking tasks and a sweep over training-set fractions. Second, in 3B-parameter language model pre-training, we find that unembedding LAR decreases slowly and stabilizes when the model generalizes, but declines sharply when overfitting is approaching. The deviation of LAR from a non-overfitting baseline also closely tracks the generalization gap. The accelerating rate of decline in LAR prior to overfitting could enable evaluation-free methods to train generalizing models — a possibility we leave open for future investigation. Across both the grokking and large-scale settings, we discover a consistent picture: lower LAR corresponds to more diffuse weight and activation distributions, which coincides with memorization, while higher LAR corresponds to more concentrated distributions and generalization. In each case, we observe that the correlation between p and q remains high throughout training, so LAR effectively tracks how far the weight and activation distributions are from uniform. Contributions. Our main contributions are as follows. • Distributional reformulation of LAR. We show that LAR can be written as the log-overlap \sum_{i}p_{i}q_{i} between a weight spectrum p and an activation spectrum q , together with equivalent covariance and correlation forms that make the metric’s behavior interpretable in terms of elementary statistical quantities. • Effective dimension via LAR. We show that when W concentrates its energy on k singular directions and activations lie along the same directions, \mathrm{LAR}=1-\tfrac{1}{2}\log_{n}k . We verify this empirically in the grokking setting by comparing the LAR-predicted dimension to k_{95} on 12 algorithmic tasks. • LAR as an overfitting diagnostic at scale. On 3B-parameter Gemma-2 models trained with Adam and Muon, we show that unembedding LAR stabilizes in the non-overfitting regime and declines sharply under overfitting, and that its deviation from a non-overfitting baseline closely tracks the generalization gap. • LAR as a proxy for spread in p and q . We show that in practice, optimization drives p and q to be highly correlated, making \mathrm{LAR} a proxy for the variance of p and q . In pre-training experiments, the top singular value of the unembedding matrix is much larger than the rest and is the primary driver of the spread in p . This makes the contribution from the top singular direction, L_{1}=1+\tfrac{1}{2}\log_{n}(p_{1}q_{1}) , an alternative proxy for the pre-overfit signature of overfitting. 2 Related Work Spectral measures of generalization. Several lines of work have connected spectral properties of weight matrices to generalization. 3 and 21 derived margin-based generalization bounds in terms of products of spectral norms, and 1 gave tighter compression-based bounds using noise stability of trained networks. 10 later compared dozens of such measures in a large-scale empirical study. Empirically, 18 propose that the spectral density of weight matrices develops heavy tails as training progresses, and use this as a data-free signal for generalization and early stopping. 31 show that the normalized effective rank of weight matrices decreases broadly during training, and that the onset of low-rank weights coincides with the transition from memorization to generalization in grokking. In contrast, LAR depends on both weights and activations and is