AGOP as Explanation: From Feature Learning to Per-Sample Attribution in Image Classifiers
Raj Kiran Gupta Katakam · 2026
Read AGOP as Explanation: From Feature Learning to Per-Sample Attribution in Image Classifiers by Raj Kiran Gupta Katakam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12816.txt.
Introduction
Katakam asks whether the Average Gradient Outer Product, which governs feature learning, can explain predictions. He introduces AGOP-Weighted, which multiplies the per-sample gradient by a training prior that suppresses noise and amplifies important pixels. Two variants follow: AGOP-Local, the per-sample gradient alone, and AGOP-Global, a zero-cost map. On benchmarks with pixel-level ground truth, AGOP-Weighted beats Integrated Gradients by 44 percent on linear tasks, and AGOP-Global beats it sevenfold on multiplicative tasks.
Catalog record
| CREATOR | Raj Kiran Gupta Katakam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24982 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12816
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AGOP as Explanation: From Feature Learning to Per-Sample Attribution in Image Classifiers Raj Kiran Gupta Katakam Abstract The Average Gradient Outer Product (AGOP) governs feature learning in neural networks: the Neural Feature Ansatz states that weight Gram matrices at each layer align with the corresponding AGOP matrices computed over the training distribution. We ask a complementary question: can this same quantity serve as a post-hoc attribution method for explaining individual predictions? We introduce AGOP - Weighted : a novel attribution method that multiplies the per-sample gradient by diag ( M ) / max diag ( M ) \sqrt{\operatorname{diag}(M)/\max\operatorname{diag}(M)} , a training-distribution prior that suppresses gradient noise and amplifies consistently important pixels—a combination not present in any prior attribution method. We formalise two companion variants—AGOP - Local (per-sample gradient, equivalent to VanillaGrad) and AGOP - Global ( diag ( M ) \operatorname{diag}(M) directly as a zero-cost saliency map)—and implement an efficient training-time accumulation hook; AGOP - Global then requires zero inference cost (disk lookup) while AGOP - Weighted requires only a single gradient pass. We conduct the first rigorous comparison of AGOP attribution against Integrated Gradients (IG), SmoothGrad, GradCAM, and VanillaGrad across two benchmarks with pixel-level ground truth: (i) the synthetic XAI-TRIS benchmark (four classification scenarios, 8 × \times 8 images, CNN8by8) and (ii) the photorealistic CLEVR-XAI benchmark (ResNet-18 fine-tuned from ImageNet). AGOP - Weighted achieves 44% higher mIoU than IG on linear tasks; AGOP - Global achieves 7 × 7\times higher mIoU than IG on multiplicative tasks (where IG falls below random) at zero inference cost. Both findings generalise to ResNet-18 on CLEVR-XAI ( + 18 % +18\% and + 37 % +37\% respectively). We further show that GradCAM fails on small-resolution images due to spatial resolution collapse, and that diag ( M ) \operatorname{diag}(M) quality improves monotonically throughout training even after classification accuracy has plateaued. [[email protected], ] keywords explainability \sep attribution \sep AGOP \sep gradient-based saliency \sep image classification \sep neural feature ansatz 1 Introduction 1.1 Motivation Explaining what a neural network “looks at” when making a prediction has become a central challenge in trustworthy machine learning. In image classification, attribution methods assign a relevance score to each input pixel, producing a saliency map that highlights which regions of the image drove the model’s decision. The field has produced a rich family of such methods: Integrated Gradients [ 15 ] uses a path integral from a baseline to the input; GradCAM [ 12 ] projects class-discriminative gradients back onto convolutional feature maps; and SmoothGrad [ 14 ] averages gradients over noise-perturbed copies of the input. These methods share a common limitation: they treat the trained model as a black box, applying geometric constructs independently of how the network learned its representations. 1.2 A Different Starting Point: AGOP Recent theoretical work has revealed that the Average Gradient Outer Product (AGOP) is not merely one post-hoc tool among many—it is the mathematical object that governs feature learning in neural networks. Radhakrishnan et al. [ 8 ] proved that the weight Gram matrices W_{l}^{\top}W_{l} at each layer align with the corresponding AGOP matrices computed over the training distribution. This is the Neural Feature Ansatz (NFA): the geometry of the network’s weights is encoded in its gradient statistics. Beaglehole et al. [ 3 ] extended this to convolutional networks, showing that patch-based AGOP recovers the filter covariance. The diagonal of the AGOP matrix, \operatorname{diag}(M) , has a natural attribution interpretation: it is the average squared partial derivative of the model output with respect to each input dimension, accumulated over the training set. Input dimensions that the model consistently relies upon receive high attribution under \operatorname{diag}(M) —precisely what a good global saliency map should capture. 1.3 Contributions 1. AGOP-Weighted: a novel attribution method that multiplies the per-sample gradient by a training-distribution prior (normalised RMS gradient w ), bridging per-sample and global attribution for the first time. While AGOP is an established theoretical construct [ 8 ] , its application as a pixel-level attribution method for image classifiers—and specifically the AGOP - Weighted formulation—is introduced here. Companion variants: AGOP - Local (equivalent to VanillaGrad, situating it within the AGOP framework) and AGOP - Global ( \operatorname{diag}(M) as a zero-cost dataset-level saliency prior, not a per-instance explanation). 2. AGOPTrainingHook : an implementation that accumulates \operatorname{diag}(M) during standard training using per-sample autograd.grad backward passes; AGOP - Global then operates at zero inference cost (disk lookup of \operatorname{diag}(M) ) while AGOP - Weighted requires a single gradient pass per input. 3. First rigorous evaluation on XAI-TRIS [ 5 ] across four classification scenarios and two background conditions, using five localization and faithfulness metrics. 4. Empirical findings : AGOP - Weighted achieves the best mIoU on linear tasks ( +44\% over IG); AGOP - Global achieves 7\times IG’s mIoU on multiplicative tasks; GradCAM fails on small-resolution images. 5. Generalisation to CLEVR-XAI [ 2 ] with ResNet-18 fine-tuned from ImageNet, confirming all core findings. 2 Related Work Gradient-based attribution. Throughout, f_{c}(x) denotes the logit for class c . VanillaGrad [ 13 ] computes |\partial f_{c}/\partial x| with a single backward pass. Integrated Gradients [ 15 ] addresses gradient saturation by integrating along a path from a reference baseline to the input, satisfying completeness and sensitivity axioms. SmoothGrad [ 14 ] reduces gradient noise by averaging over n noise-perturbed copies of the input. GradCAM [ 12 ] and GradCAM++ [ 4 ] weight convolutional feature maps by gradient-based channel importance and upsample to input resolution. AGOP, NFA, and recursive feature machines. Radhakrishnan et al. [ 8 ] proved the NFA for fully-connected networks trained with gradient descent. Beaglehole et al. [ 3 ] extended the result to convolutional networks via patch-based AGOP. The xRFM framework [ 9 ] uses the diagonal of the AGOP matrix for coordinate-level feature importance in tabular settings but does not compare against pixel-level XAI baselines on image benchmarks. Our work fills this gap. Perturbation-based and global methods. LIME [ 10 ] and SHAP [ 7 ] treat the model as a black box, fitting surrogate models or computing Shapley values on perturbed samples; both are complementary to gradient-based methods but more expensive at inference. Most attribution methods are local —producing a separate map per input; AGOP - Global is global , acting as a dataset-level diagnostic of learned features, not a per-instance explanation. XAI benchmarks. XAI-TRIS [ 5 ] provides 8 \times 8 synthetic images with deterministic pixel masks across four task types of increasing non-linearity. CLEVR-XAI [ 2 ] provides photorealistic 3D scenes with per-question ground-truth object masks. OpenXAI [ 1 ] provides a tabular XAI evaluation framework; Samek et al. [ 11 ] establish the deletion/insertion faithfulness protocol we adopt. 3 Attribution Methods We evaluate five standard baselines and three AGOP variants, plus a random attribution baseline for reference. All methods produce a non-negative saliency map A\in\mathbb{R}^{H\times W} for a given input image x\in\mathbb{R}^{C\times H\times W} and predicted class c . 3.1 Standard Baselines VanillaGrad [ 13 ] . \text{attr}_{\text{VG}}(h,w)=\sum_{ch}\left|\partial f_{\hat{c}}(x)/\partial x[ch,h,w]\right| , where ch indexes colour channels and \hat{c} is the predicted class. Computed with a single backward pass ( 1\times cost). Integrated Gradients (IG) [ 15 ] . \text{IG}_{j}(x)=(x_{j}-x^{\prime}_{j})\cdot\frac{1}{T}\sum_{k=1}^{T}\frac{\partial f_{c}(x^{\prime}+\frac{k}{T}(x-x^{\prime}))}{\partial x_{j}} , where j indexes input dimensions,