Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositions
Thomas Vitry, Kieran Edgeworth, Stefan Wermter, Jae Hee Lee · 2026
Read Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositions by Thomas Vitry, Kieran Edgeworth, Stefan Wermter, Jae Hee Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28780.txt.
Introduction
Thomas Vitry, Kieran Edgeworth, Stefan Wermter, and Jae Hee Lee present a label-free method for finding spurious concepts in frozen models. Per target class they gather patches from inputs of that class and factor activations into interpretable concept vectors. Candidate concepts rank by an estimator from backpropagated gradients on misclassified cases: bias concepts fire correcting false negatives and quiet correcting false positives. On Colored MNIST and Waterbirds it recovers the spurious cue, and suppressing concepts at inference lifts worst-group accuracy by up to 17.9 points on Waterbirds without retraining.
Catalog record
| CREATOR | Thomas Vitry, Kieran Edgeworth, Stefan Wermter, Jae Hee Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55411 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositions Thomas Vitry, Kieran Edgeworth, Stefan Wermter, Jae Hee Lee Abstract Vision classifiers can exploit spurious correlations, achieving high in-distribution accuracy yet failing under distribution shift. Existing approaches to bias mitigation and analysis often depend on curated datasets, spurious-attribute or group labels, or retraining, which may be infeasible once a model is deployed or the relevant bias is unknown. We present a bias-label-free, post-hoc method for identifying spurious concepts in frozen vision models, relying only on standard class labels from a held-out audit dataset. For each target class, we collect patches from inputs predicted as that class and apply non-negative matrix factorization to intermediate activations to obtain a bank of interpretable concept vectors. Candidate concepts are then ranked with a bias estimator derived from their interaction with backpropagated gradients on misclassified examples: bias concepts tend to get activated when correcting false negatives and suppressed when correcting false positives. On Colored MNIST and Waterbirds the method recovers concepts aligned with the known spurious cue, and on CelebA it surfaces decision-relevant directions that only partially coincide with the annotated gender attribute; suppressing the top-ranked concepts at inference time improves worst-group accuracy by up to 17.9 17.9 percentage points on Waterbirds and 10.4 10.4 on CelebA without any retraining or parameter updates. Our method identifies decision-relevant spurious directions that need not coincide with annotated ones, providing both an interpretable auditing tool and an actionable debiasing handle for frozen vision models. Code is available at https://github.com/vitryt/label-free-bias-identification . * * footnotetext: Corresponding authors 1 Introduction Modern vision models can achieve high in-distribution accuracy while relying on spurious correlations: attributes that predict the label in the training data but are not causally related to the task. This shortcut learning behavior [ 12 ] can lead to brittle generalization and systematic failures when the correlation shifts at deployment. For example, in fine-grained recognition the model may over-rely on context: on Waterbirds [ 39 ] , background type is spuriously correlated with the bird label during training. Such shortcuts may remain hidden when the available held-out set (e.g., the validation set) shares the same correlation, yet cause systematic errors when the correlation breaks at deployment. To counter shortcut learning, methods often mitigate spurious correlations during data collection or training [ 29 ] . When the relevant bias concept is known, it can be targeted for mitigation via retraining [ 5 , 9 , 10 , 22 ] or post-hoc interventions on inputs or representations [ 2 ] . These pipelines typically assume that the spurious attribute has already been identified and therefore do not solve the discovery problem. In practice, however, we only have access to a deployed frozen model and a held-out set, but not to the original training pipeline, and rarely to bias or group annotations. In this setting, analyzing a deployed model is difficult because the spurious attribute is often unknown, and collecting new annotations can be costly. This motivates methods that can propose plausible spurious attributes post-hoc, while returning a concept vector in the latent space that is directly compatible with downstream interventions. Figure 1 : Diagram of the bias identification method. (a) We collect false negatives and false positives from a held-out set (the bias-audit set) and pass them through the model. Here, studying class y{=}0 (landbird), the sample is a false positive: a waterbird misclassified as a landbird. (b) Using non-negative concept decomposition trained beforehand on the bias-audit set, we decompose activations of the sample into its concept coefficients. (c) Using the ground-truth label from the bias-audit set, we backpropagate the gradient to the activations. (d) We decompose the updated activations (after the gradient step) using the same concept bank. (e) We compare which concepts are activated before and after the gradient step. Concepts that are removed (for false positives) or added (for false negatives) by the gradient step are candidate bias concepts. (f) The identified bias concepts come with their encoding in the activation space and their most activating patches, enabling interpretation. Note that bias concepts are identified by aggregating over multiple samples, not from a single example. We study whether unsupervised, representation-level concept discovery can detect spurious attributes in frozen vision models when bias annotations are unavailable. Our approach decomposes intermediate activations into non-negative concept vectors and uses a held-out set (without group or bias labels) to probe the model via gradients ( Section 3 ). We first test whether the spurious attribute learned by a biased model reliably appears as a distinct concept direction in this basis ( Section 4.1 ). Building on this, we propose a bias identification criterion based on how concept activations change under a gradient descent step: bias concepts tend to be activated when correcting false negatives and suppressed when correcting false positives, while concepts encoding intrinsic class evidence change much less asymmetrically. We validate the scoring criterion against ground-truth bias ( Sections 4.2 and 4.3 ) and then test whether the identified concepts are actionable, by suppressing them at inference time without any parameter updates ( Section 4.4 ). We evaluate these on Colored MNIST, Waterbirds and CelebA, three standard benchmarks for spurious correlation learning, and find that concept suppression substantially improves worst-group accuracy on Waterbirds and CelebA. The resulting approach provides both an interpretable auditing tool and a concrete debiasing handle for frozen vision models when bias annotations and retraining are unavailable. 2 Preliminaries We consider a multi-class classification task with inputs x\in\mathcal{X}\subseteq\mathbb{R}^{d} and labels y\in\mathcal{Y}=\{1,\dots,C\} , and a hypothetical ideal classifier f^{*}:\mathcal{X}\to\mathcal{Y} . Following [ 29 ] , an attribute is a function b:\mathcal{X}\to\{0,1\} ; it is intrinsic to class y if it defines membership (e.g., digit shape), and a bias attribute for y if it is spuriously correlated with y without being intrinsic. We are given a frozen classifier f with class scores f(x)=(f_{c}(x))_{c=1}^{C} and a held-out bias-audit set \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n} with y_{i}=f^{*}(x_{i}) but no attribute or group annotations. \mathcal{D} need not be from the training set and may be unbiased or distribution-shifted; it may come from a validation split, relabeled deployment data, or targeted stress tests. Our goal is to detect, for each class y , concept directions in the representation space that plausibly encode bias attributes used by the model and that are empirically implicated in its decisions. Non-Negative Concept Decomposition. Following CRAFT [ 11 ] , we write f=h\circ g where g:\mathcal{X}\to\mathcal{A} maps an input to an intermediate representation a=g(x)\in\mathcal{A}\subseteq\mathbb{R}^{p} and h:\mathcal{A}\to\mathbb{R}^{C} maps the representation to class scores. We choose a representation layer after a ReLU so that a\in\mathbb{R}^{p}_{\geq 0} , enabling non-negative matrix factorization (NMF). For each class y , let \hat{y}(x)=\arg\max_{c\in\mathcal{Y}}f_{c}(x) be the predicted label and collect inputs predicted as y : \mathcal{X}_{y}=\{x_{i}:\hat{y}(x_{i})=y\}. A patch dataset is then created by applying a crop-and-resize operator \pi_{s} (patch size s ) to these images, yielding patches \mathcal{P}_{y}=\{\pi_{s}(x):x\in\mathcal{X}_{y}\} . Let A_{y}=g(\mathcal{P}_{y})\in\mathbb{R}_{\geq 0}^{n_{y}\times p} be the resulting activations. We compute a class-conditional concept bank (U_{y},W_{y})=\underset{U_{y}\geq 0,\,W_{y}\geq 0}{\text{arg}\,\text{min}}\frac{1}{2}\left\lVert A_{y}-U_{y}W_{y}^{\top}\right\rVert_{F}^{2}, (1) via NMF, where W_{y}\in\mathbb{R}_{\geq 0}^{p\times r} contains r concept vectors w_{y,1},\dots,w_{y,r} and U_{y}\in\mathbb{R}_{\geq 0}^{n_{y}\times r} the corresponding non-negative concept coefficients. As in [ 11 ] , this typically yields semantically meaningful, relatively disentangled concepts; the decomposition is approximate and