Gradient Extrapolation for Debiased Representation Learning
Ihab Asaad, Maha Shadaydeh, Joachim Denzler · 2025
Read Gradient Extrapolation for Debiased Representation Learning by Ihab Asaad, Maha Shadaydeh, Joachim Denzler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13236.txt.
Introduction
Ihab Asaad, Maha Shadaydeh, and Joachim Denzler propose in 2025 gradient extrapolation to purge bias from learned features. Classifiers often lean on spurious links that vanish at test time. Their method trains on two batches with unlike amounts of the spurious link.
The target gradient stretches the line between the two batch gradients. When the stretch aims toward the cleaner batch, training sheds the bias. Classic training and resampling fall out as special cases of the frame. Six benchmarks show scores that match or beat top debias tools. Bounds on the stretch factor tell users how far to push it.
Catalog record
| CREATOR | Ihab Asaad, Maha Shadaydeh, Joachim Denzler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13236
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gradient Extrapolation for Debiased Representation Learning Ihab Asaad, Maha Shadaydeh, Joachim Denzler Abstract Machine learning classification models trained with empirical risk minimization (ERM) often inadvertently rely on spurious correlations. When absent in the test data, these unintended associations between non-target attributes and target labels lead to poor generalization. This paper addresses this problem from a model optimization perspective and proposes a novel method, G radient E xtrapolation for Debiased R epresentatio n L e arning (GERNE), designed to learn debiased representations in both known and unknown attribute training cases. GERNE uses two distinct batches with different amounts of spurious correlations and defines the target gradient as a linear extrapolation of the gradients computed from each batch’s loss. Our analysis shows that when the extrapolated gradient points toward the batch gradient with fewer spurious correlations, it effectively guides training toward learning a debiased model. GERNE serves as a general framework for debiasing, encompassing ERM and Resampling methods as special cases. We derive the theoretical upper and lower bounds of the extrapolation factor employed by GERNE. By tuning this factor, GERNE can adapt to maximize either Group-Balanced Accuracy (GBA) or Worst-Group Accuracy (WGA). We validate GERNE on five vision and one NLP benchmarks, demonstrating competitive and often superior performance compared to state-of-the-art baselines. The project page is available at: https://gerne-debias.github.io/ . 1 Introduction Deep learning models have demonstrated significant success in various classification tasks, but their performance is often compromised by datasets containing prevalent spurious correlations in the majority of samples [ 52 , 29 , 18 , 13 ] . Spurious correlations refer to unintended associations between easy-to-learn, non-target attributes and target labels. These correlations often cause models trained with Empirical Risk Minimization (ERM) [ 45 ] to rely on these correlations instead of the true, intrinsic features of the classes [ 12 , 10 , 40 ] . This occurs because the ERM objective optimizes for the average performance [ 45 ] , thereby biasing the model toward the easy-to-learn features that are predictive for the majority of training samples. As a result, ERM-trained models often exhibit poor generalization when these spurious features are absent in the test data. For instance, in the Waterbirds classification task [ 46 ] , where the goal is to classify a bird as either a waterbird or a landbird, the majority of waterbirds are associated with water backgrounds. In contrast, the majority of landbirds are associated with land backgrounds. A model trained with ERM might learn to classify the birds based on the background—water for waterbirds and land for landbirds—rather than focusing on the birds’ intrinsic characteristics. This reliance on the spurious feature allows the model to perform well on the majority of training samples, where these correlations hold, but fails to generalize to test samples where these correlations are absent (e.g., waterbirds on land). Examples of Waterbirds images shown in Fig. 1(a) . Avoiding spurious correlations is crucial across various applications, including medical imaging [ 25 , 37 ] , finance [ 11 ] , and climate modeling [ 17 ] . This pervasive challenge has spurred extensive research into strategies for mitigating the negative effects of spurious correlations, particularly under varying levels of attribute information availability. The authors of [ 50 ] provide a comprehensive review of the methods and research directions aimed at addressing this issue. In an ideal scenario, where attribute information is available in both the training and validation sets, methods can leverage this information to counteract spurious correlations [ 16 , 39 , 48 ] . When attribute information is available only in the validation set, methods either incorporate this set into the training process [ 18 , 32 , 42 ] or restrict its use to model selection and hyperparameter tuning [ 31 , 28 , 29 , 27 , 35 ] . Despite these efforts, existing methods still struggle to fully avoid learning spurious correlations, especially when the number of samples without spurious correlations is very limited in the training dataset, leading to poor generalization on the test data where these correlations are absent. In this paper, we adopt a different research approach, seeking to address the issue of spurious correlations from a model optimization perspective. We propose a novel method, Gradient Extrapolation for Debiased Representation Learning (GERNE), to avoid reliance on spurious features and learn debiased representations. The core idea is to sample two types of batches with varying amounts of spurious correlations ( Fig. 1(b) ) and compute the two losses on these two batches. We assume that the difference between the gradients of these losses captures a debiasing direction. Therefore, we define our target gradient as the linear extrapolation of these two gradients toward the gradient of the batch with fewer amount of spurious correlations ( Fig. 1(c) ). The contributions of this paper can be summarized as follows: • We propose GERNE as a general framework for debiasing, with methods such as ERM and Resampling being shown as special cases. • We derive the theoretical upper and lower bounds of the extrapolation factor and establish a direct connection between the extrapolation factor and the risk for the worst-case group. We show that tuning this factor within these bounds enables GERNE to adaptively optimize for either Group-Balanced Accuracy or Worst-Group Accuracy. • We validate our approach on six benchmarks spanning both vision and NLP tasks, under both known and unknown attribute cases, demonstrating competitive and often superior performance compared to state-of-the-art methods—particularly in scenarios where samples without spurious correlations are scarce. (a) (b) (c) Figure 1 : 1(a) : Sample images from the waterbirds classification task. Most landbird images appear with land backgrounds (i.e., y=1,a=1 ), while most waterbird images appear with water backgrounds (i.e., y=2,a=2 ). This correlation between bird class and background introduces spurious correlations in the dataset. 1(b) : Visualization of batch construction. B_{b} refers to a biased batch where the majority of images from class y=1 (top row) have attribute a=1 (yellow), and the majority of images from class y=2 (bottom row) have attribute a=2 (light-blue). B_{lb} represents a less biased batch, with a more balanced attribute distribution within each class, controlled by c (here c=\frac{1}{2} ). B_{rs} depicts a batch with group-balanced distribution and refers to the batch used in the Resampling method [ 16 ] . B_{ext} simulates GERNE’s batch with c\cdot(\beta+1)>1 , where the dataset’s minority groups appear as majorities in the batch. 1(c) : A simplified 2D representation of gradient extrapolation where \theta\in\mathbb{R}^{2} . \nabla_{\theta}\mathcal{L}_{b} is the gradient computed on B_{b} ; training with this gradient is equivalent to training with the ERM objective. \nabla_{\theta}\mathcal{L}_{lb} represents the gradient computed on B_{lb} . \nabla_{\theta}\mathcal{L}_{rs} is the gradient computed on B_{rs} , which is equivalent, in expectation, to an extrapolated gradient with c\cdot(\beta+1)=1 . Finally, \nabla_{\theta}\mathcal{L}_{ext} is our extrapolated gradient, with the extrapolation factor \beta modulating the degree of debiasing based on the strength of spurious correlations present in the dataset. 2 Related Work Debiasing according to attribute annotations availability. Numerous studies have leveraged attribute annotations to mitigate spurious correlations and learning debiased representation [ 39 , 55 , 3 , 51 , 53 ] . For instance, Group DRO [ 39 ] optimizes model performance on