Learn to Rank: Visual Attribution by Learning Importance Ranking
David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch, Samuel Schulter, Horst Possegger · 2026
Read Learn to Rank: Visual Attribution by Learning Importance Ranking by David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch, Samuel Schulter, Horst Possegger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05819.txt.
Introduction
Schinagl and four colleagues learn visual attribution maps by optimizing deletion and insertion scores head-on. Treating ranking as permutation learning with a Gumbel-Sinkhorn relaxation enables end-to-end training via attribution-guided tweaks. Inference yields dense pixel-level maps in one forward pass, sharper and boundary-aligned, above all for vision transformers.
Catalog record
| CREATOR | David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch, Samuel Schulter, Horst Possegger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71998 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learn to Rank: Visual Attribution by Learning Importance Ranking David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch, Samuel Schulter, Horst Possegger Abstract Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model’s prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models. 1 Introduction Over the past decade, deep neural networks [ 25 , 32 , 40 , 41 ] have powered the vast majority of vision models. The growing complexity of these inherently opaque models, however, makes interpreting their decisions difficult. Especially in safety-critical domains such as healthcare or autonomous driving, explaining why a model predicts a certain outcome can be as crucial as the accuracy of the prediction itself. A well-established approach for explainability in computer vision is the creation of visual attribution maps, which highlight the input regions most relevant to a model’s prediction [ 69 , 53 , 56 , 47 , 14 ] . Such attribution maps provide an intuitive, human-interpretable way to understand model behavior and can help identify potential biases or failure modes [ 11 , 4 ] . The majority of attribution methods are post-hoc , i.e . they explain already trained models without modifying their architecture. These can be broadly categorized into three groups: Propagation-based approaches which rely on propagating internal signals through the target model, such as gradients [ 56 , 69 , 62 ] , activations [ 71 , 53 , 65 ] or attention weights [ 1 , 14 , 15 ] in transformer-based vision models. They are very efficient and align with the model’s internal computations but are architecture-specific, can bias toward low-level features like edges [ 2 , 59 ] and may under-represent non-attention pathways in transformer models. In contrast, perturbation-based methods [ 47 , 51 , 42 , 30 ] infer feature relevance from changes in the target model’s output when parts of the input are modified. They provide more causally grounded attributions, as they directly measure the effects of input changes, including higher-order interactions. However, they are computationally demanding, requiring many model evaluations to produce a single attribution map. For transformer-based vision models, they usually operate on the native patch representation [ 64 , 66 , 27 ] yielding patch-level explanations that must be upsampled to pixel space, resulting in coarse, spatially imprecise maps, as shown in Figure 1 . The third and least explored category, learning-based (amortized) explainers shift the computational burden to an offline training phase by learning a separate explainer model, which is then able to generate attribution maps very efficiently during inference [ 21 ] . However, the current training strategies typically rely on either supervision from other attribution methods [ 58 , 33 ] and may adopt their biases, or on self-supervised surrogate objectives that are based on restrictive assumptions like a binary signal to noise decomposition of features [ 18 , 7 ] . Motivated by these limitations, it is natural to ask whether we can avoid attribution heuristics and instead optimize attribution quality directly . Common and widely-used attribution metrics are the Deletion and Insertion metrics [ 47 ] . They quantify the change in a model’s output when features are removed or added back in order of importance. Recent metric-driven approaches [ 20 , 64 ] for vision transformers pursue this direction, but do so via a perturbation-based, per-sample optimization on the patch representation. Consequently, they retain the high computational cost and cannot refine the coarse spatial resolution. In this work, we propose Amortized Hybrid Attribution (AHA) : a learning-based approach that bridges these families of attribution methods by integrating their complementary strengths into a unified framework. We retain the causal grounding of perturbation-based methods by learning from the target model’s outputs under attribution-guided perturbations. Since we couple explainer and target model through end-to-end backpropagation, we inherit the model-consistent attribution characteristic of propagation-based approaches. We realize this by directly optimizing the Deletion and Insertion metrics. To overcome the non-differentiability of these metrics, caused by the sorting operations they depend on, we leverage the observation that they depend solely on the order of pixel importance rather than their exact values, and treat them as permutation problems. By replacing the hard sorting and ranking operations with smooth approximations, we enable end-to-end training of the explainer model. At inference, we preserve the amortized efficiency of learned explainers (single forward pass), but we also keep the door open to the propagation-based regime: we can optionally perform a few additional backward steps through the target model to locally refine the explainer’s output on a specific sample. This effectively combines a learned, metric-aligned prior with a cheap, model-specific propagation signal. Due to the end-to-end optimization our explainer is able to generate dense attributions that align closely with object boundaries and visually meaningful structures, even for transformer-based vision models, as shown in Figure 1 . Original [0] AHA ( T=0 ) [0] Bi-Att. [ 17 ] MDA [ 64 ] LeGrad [ 10 ] TIS [ 27 ] ViT-CX [ 66 ] cairn (Norwich terrier) can opener (bottlecap) impala (baboon) reel Figure 1 : Qualitative comparison of attribution maps produced by our method (AHA) and recent ViT attribution methods for a ViT-B/16 classifier [ 25 ] on ImageNet [ 22 ] samples. The green / red labels mark correct / incorrect predictions (ground truth in parentheses). While existing approaches operate on the patch representation and produce coarse, blocky attribution maps, our method generates dense, pixel-level attributions that align closely with object boundaries. Note that in this case no additional backprop refinement steps were applied at inference (denoted T=0 ). Our contributions can be summarized as follows: • We propose a learning-based attribution framework that bridges perturbation- and propagation-based explainability. • Our amortized explainer directly optimizes differentiable surrogates of Deletion and Insertion , producing dense pixel-space attributions also for transformer-based vision models. • We formulate metric optimization as a permutation learning problem and enable end-to-end training via a Gumbel-Sinkhorn relaxation of sorting and top- k selection. • To reduce the computational burden and mitigate pixel-level shortcut artifacts, the training procedure leverages region-level permutation, perturbation-step sampling, and grid augmentation. • At inference, the method retains single-pass efficiency and optionally adds a few backprop refinement steps for sample-specific adjustments. 2 Related Work Visual attribution methods highlight the most influential input regions for a