Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks
Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis · 2025
Read Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks by Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25480.txt.
Introduction
Florian A Holzl, Daniel Rueckert, and Georgios Kaissis present Gradient-Weight Matching. Their 2025 paper seeks a train-time proxy for generalization. Robust validation metrics are needed in deep learning. They detect overfitting and poor generalization during training. Training changes also need monitoring for model quality. Supervised classification is their certain setting here.
Gradient-Weight Matching quantifies coherence between gradients and weights. Per-sample gradients are compared with current model weights. Effective learning corresponds to coherent matching, they show. Misalignment indicates deteriorating generalization in their tests. The metric is quickly computable during training runs. It reflects sample-certain contributions to learning changes. Dataset-wide learning patterns are also captured simultaneously. Extensive tests show it predicts optimal early stopping. Principled model comparisons are enabled by the metric. Influential training samples can be identified with it. No separate validation set is needed for study. That offers a useful tool for training monitoring.
Catalog record
| CREATOR | Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25480
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis Abstract Robust validation metrics remain essential in contemporary deep learning, not only to detect overfitting and poor generalization, but also to monitor training dynamics. In the supervised classification setting, we investigate whether interactions between training data and model weights can yield such a metric that both tracks generalization during training and attributes performance to individual training samples. We introduce Gradient-Weight Alignment (GWA), quantifying the coherence between per-sample gradients and model weights. We show that effective learning corresponds to coherent alignment, while misalignment indicates deteriorating generalization. GWA is efficiently computable during training and reflects both sample-specific contributions and dataset-wide learning dynamics. Extensive experiments show that GWA accurately predicts optimal early stopping, enables principled model comparisons, and identifies influential training samples, providing a validation-set-free approach for model analysis directly from the training data. 1 Introduction Despite the recent surge in self-supervised learning, optimization with cross-entropy loss remains dominant in modern deep learning for both supervised training and fine-tuning. Its enduring popularity stems from its strong label-based learning signal and implicit modeling of predictions as probabilities. However, this very strength – relying on maximum likelihood estimation – introduces vulnerabilities to overconfidence and sensitivity to noise in both input data and labels. Diagnosing these issues typically relies on hold-out validation sets, which operate under the assumption of independent and identically distributed ( i.i.d. ) data. While effective, this standard approach necessitates labeled data that is rendered unavailable for training and offers limited insight into how these issues can be attributed to training set samples. This motivates a critical question: can we effectively assess model generalization and diagnose potential problems solely using information available during training? Prior work indicates that robust generalization emerges when all training samples contribute coherently towards a shared learning goal, that is, their gradients are directionally well aligned [ 1 , 2 ] . Conversely, conflicting or misaligned gradient directions indicate a potential failure to generalize. However, computing pairwise gradient alignment is highly memory-intensive and is not an inherently distributional quantity; in other words, the average gradient alignment over the dataset provides minimal insight into individual samples’ contribution to training. In this work, we thus turn to the alignment between per-sample gradients and the model weights as a measure for estimating key training dynamics and predicting generalization in training with cross-entropy. This quantity, which we refer to as Gradient-Weight Alignment (GWA) , as well as its distribution, capture the degree of gradient coherence across the dataset and allow linking model performance directly to the individual underlying input-label pairs ( Fig. 1 ). Moreover, we show that GWA can be efficiently estimated during training (online), providing insights into training dynamics even during large-scale optimization with negligible overhead. We argue through extensive empirical evaluation that GWA accurately identifies the point in training beyond which further training steps yield diminishing returns in terms of generalization performance, even rendering held-out validation sets redundant. Our contributions can be summarized as follows: • We introduce GWA as a novel proxy for generalization performance during training - effectively replacing the need for withholding a separate validation set. • GWA reveals the influence of individual training samples on optimization, providing a powerful diagnostic tool for understanding data quality issues like outliers and label errors. • We show GWA scales to large-scale training and finetuning, and remains robust under input/label noise, offering a superior criterion for choosing models deployed in practice. 2 Related Work Figure 1: Gradient alignment among individual samples \nabla\mathcal{L}_{i} as well as the model weights varies during training, with coherent per-sample gradient direction reflecting generalization. Line plots illustrate how GWA captures gradient coherence and model performance at different time points t . Quantifying the generalization gap - understanding how well a model performs on unseen data - is a central challenge in deep learning [ 3 ] . Early approaches without hold-out data focused on estimating this gap using the curvature of the loss function, offering insights into sample influence. However, computing second-order derivatives is computationally expensive [ 4 , 5 , 6 , 7 ] , and curvature can be unstable during training and sensitive to hyperparameters [ 8 , 9 , 10 ] . More recently, influence functions [ 11 ] and sub-sampling estimators [ 12 , 13 ] have emerged to compute per-sample influence at the end of training without second-order derivatives. Our work differs by focusing on an efficient train-time estimator of generalization for online assessment rather than post-training influence analysis. We contend that comprehensively evaluating and characterizing model dynamics is crucial for understanding optimization and its challenges - such as potential overfitting. Prior work has observed that Stochastic Gradient Descent (SGD) exhibits a “simplicity bias”, initially learning simple patterns before fitting increasingly complex functions [ 14 , 15 , 16 , 17 , 18 , 19 ] . While we build on these insights, we move beyond characterizing what is learned to quantifying when further learning yields diminishing returns for generalization – a relatively underexplored area outside the noisy label regime [ 20 , 21 ] . Most notably from this area, we compare against the recent LabelWave [ 22 , 23 ] which quantifies model prediction changes to determine a suitable early stopping point. Prediction-change-focused methods like LabelWave are designed primarily for early stopping in noisy optimization, but offer limited insight into underlying training dynamics. In contrast, our GWA directly reflects these dynamics via a sample-level measure. Quantifying the coherence of per-sample gradients captures training dynamics such as faster convergence and improved generalization [ 24 , 1 , 25 , 26 , 27 ] . As illustrated in Fig. 1 , this coherence is captured by the direction of the gradients, a signal distinct from their magnitude. However, computing coherence in the aforementioned approaches requires the gradients of all training samples to be stored in memory – an impractical limitation for large datasets – and often provides only aggregate statistics, obscuring valuable per-sample insights. Gradient Disparity (GD) [ 28 ] provides a more efficient approach for k -fold cross validation but has not been shown to work in large scale settings. Our approach is scalable and addresses these limitations by leveraging the model weights as a reference vector. Theoretical work demonstrates that under ideal conditions -perfectly classifiable data- weights converge in direction, i.e. , maintain a specific direction while the gradient aligns with the weights’ direction [ 29 , 2 ] . While the impact of noise in this case is still unexplored, some levels of noise can be beneficial [ 30 , 31 ] , and recent empirical research affirms directional stability even with stochasticity by analyzing batch gradient direction relative to the optimal weights [ 32 ] . Our proposed GWA specifically focuses on quantifying alignment in the presence of realistic data variance. Moving from pairwise and aggregate gradient statistics to GWA is not only more efficient but allows us to link individual samples to generalization performance in stochastic optimization. In the following, we propose an efficient computation strategy of per-sample gradient-weight alignment and leverage the properties of the resulting alignment distribution as a novel measure of generalization and sample-level influence.