Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization
Shahin Hakemi, Naveed Akhtar, Ghulam Mubashar Hassan, Ajmal Mian · 2025
Read Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization by Shahin Hakemi, Naveed Akhtar, Ghulam Mubashar Hassan, Ajmal Mian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14182.txt.
Introduction
Shahin Hakemi and co-authors fix a trained model by editing one weight in a 2025 paper. Networks love shortcuts: the simplest feature that predicts the label wins, even when the link is a fluke. Fixing that during training costs time, and the flaw often shows only after deployment. Spuriousness is also a matter of degree, not a yes or no trait.
Their method asks how spurious a feature is, then distracts the model in proportion. The fix lands after training, on a single weight, avoiding a fresh training run. A cheap post hoc patch suits deployed models whose owners cannot retrain at will. The paper reframes debiasing as careful editing rather than rebuilding.
Catalog record
| CREATOR | Shahin Hakemi, Naveed Akhtar, Ghulam Mubashar Hassan, Ajmal Mian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization Shahin Hakemi, Naveed Akhtar, Ghulam Mubashar Hassan, Ajmal Mian Abstract \justifying Neural network training tends to exploit the simplest features as shortcuts to greedily minimize training loss. However, some of these features might be spuriously correlated with the target labels, leading to incorrect predictions by the model. Several methods have been proposed to address this issue. Focusing on suppressing the spurious correlations with model training, they not only incur additional training cost, but also have limited practical utility as the model misbehavior due to spurious relations is usually discovered after its deployment. It is also often overlooked that spuriousness is a subjective notion. Hence, the precise questions that must be investigated are; to what degree a feature is spurious, and how we can proportionally distract the model’s attention from it for reliable prediction. To this end, we propose a method that enables post-hoc neutralization of spurious feature impact, controllable to an arbitrary degree. We conceptualize spurious features as fictitious sub-classes within the original classes, which can be eliminated by a class removal scheme. We then propose a unique precise class removal technique that makes a single-weight modification, which entails negligible performance compromise for the remaining classes. We perform extensive experiments, demonstrating that by editing just a single weight in a post-hoc manner, our method achieves highly competitive, or better performance against the state-of-the-art methods. Our implementation will be made public after acceptance. I Introduction Artificial Neural Networks (ANNs) that employ Empirical Risk Minimization (ERM) [ 1 ] are prone to correlate spurious features to target labels [ 2 , 3 , 4 , 5 ] . Residing in training data, such features often provide shortcuts to minimize loss, causing over-reliance of the model on them for inference [ 6 , 7 , 8 , 9 ] . This leads to poor model generalization. Currently, the prevalent paradigm of suppressing learning of spurious feature-target label correlation - aka spurious correlation - is robust model learning [ 10 , 11 , 12 , 13 , 14 ] , which either requires a subsequent model retraining [ 15 , 16 ] , or training the model robustly right from scratch [ 17 , 18 , 19 , 20 , 21 ] . Fig. 1: Illustration of the adopted fictitious class perspective. Top: A non-robust Empirical Risk Minimization (ERM) classifier may rely on incorrect set of high-level features - fictitious sub-classes within a class - to mis-associate male gender to non-blond hair. Bottom: Removing the undesired fictitious (sub-)class from the set enables robust classification. Fig. 2: (A) To remove a specific class, we first compute Specific Class Associate (SCA) score - a metric defined in this work jointly over the activations, gradients and entropy of the neural connections. The SCA score identifies the most important edge (red) for the class under consideration. We replace the weight of that edge with an analytically computed value such that the resulting decision hyperplane becomes orthogonal to its original state around the axis defined by the weight. (B) Any arbitrary sub-class for a model \mathcal{M} can be removed by making a copy \mathcal{M^{\prime}} of the model and let it classify that sub-class as one of its main classes while all its parameters, except the last layer weights, are frozen. The sub-class can be removed from \mathcal{M} with the same editing as that required for removing it from \mathcal{M^{\prime}} . (C) Spurious features can be treated as fictitious sub-classes and removed using the same process as described in (B). In any case, existing techniques deal with spurious correlation suppression in an ante-hoc manner. Leaving alone the viability and computational overhead of model retraining at the user’s end; where the model misbehavior due to spurious correlation is often first manifested, ante-hoc approaches may inadvertently compromise the overall model performance. Hence, they have limited practical value. Kirichenko et al. [ 22 ] showed that non-robust models also learn core/robust data features, albeit they lack strong reliance on them in decision making. This observation inspires us to retain the original learning of the model, thereby focusing on the possibility of post-hoc spurious correlation suppression by distracting a pre-trained model from paying too much attention to the spurious features. Another intriguing insight in the literature is presented by Eastwood et al. [ 23 ] , who argue that so-called spurious features are not entirely harmful. Aligned with [ 23 ] , we posit that some apparent spurious correlations may even help model generalization, provided the right underlying data distribution. For instance, (spuriously) correlating a seagull with the background of sea might help correctly recognizing a bird over a sea as a seagull in a typical natural image setting. Clearly, spuriousness is a subjective notion, and techniques addressing spurious correlation should provide the ability to control the extent to which it can be neutralized for optimal model performance. Unfortunately, existing methods generally fail to explicitly account for such a control. In this work, we propose a post-hoc technique that enables neutralizing the contribution of a high-level feature to model prediction by an arbitrary degree. We conceptualize high-level features as fictitious sub-classes within the original class. Our method withdraws model attention from a selected fictitious sub-class to control the model behavior (see Fig. 1 ). Building on a strong theoretical foundation, we introduce a single-weight editing method to unlearn a fictitious class such that our post-hoc editing is applied to the network connection that contributes the most to the original model behavior for that class. To find that connection, we analyze class-specific activations and gradients of the model for the given class. In addition to being a unique post-hoc method to address spurious correlation, our technique also does not require group-annotated samples for editing, as often required by some existing methods [ 24 , 17 , 22 ] . Our main contributions are summarized below. 1. We propose a first-of-its-kind post-hoc model unlearning technique to address spurious correlations. Our method edits only a single model weight to break off prediction reliance on irrelevant high-level features in the input. 2. We provide theoretical foundations leveraging class activations and model gradients to single out the most significant model weight contributing to its behavior for a given (sub-)class. Editing this weight unlearns the target class with none-to-negligible negative impact on the model performance for the remaining classes. 3. Our method operates without group label annotations and, through extensive experiments, demonstrates state-of-the-art or comparable performance in mitigating spurious correlations while being the only post-hoc method. II Related Work We discuss the key existing works by organizing them according to the aspects in which they relate to our approach. Spurious Correlation Mitigation: To mitigate spurious correlation, early methods employed distributionally robust optimization (DRO), which utilizes group annotations to up-weight the worst-group loss during optimization [ 24 , 25 , 26 , 27 , 28 ] . More recently, along similar lines, Deng et al. [ 17 ] proposed robust model learning with progressive data expansion. Though effective in terms of achieved worst-group performance, its strong dependence on group annotations limits its practicality. To mitigate the issue, other works proposed using only a limited amount of group annotated data [ 29 , 11