A Meaningful Perturbation Metric for Evaluating Explainability Methods

Danielle Cohen, Hila Chefer, Lior Wolf · 2025

Read A Meaningful Perturbation Metric for Evaluating Explainability Methods by Danielle Cohen, Hila Chefer, Lior Wolf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06800.txt.

Introduction

Danielle Cohen, Hila Chefer, and Lior Wolf proposed a fairer way to judge the heat maps that claim to explain a neural network's decisions. Different explanation methods paint wildly different maps for the same image, so the field needs trustworthy scoring. The standard test erases the pixels a map calls important, but crude erasing wrecks the image and muddies the verdict.

Their test instead repaints only the important pixels with a generative image model, keeping the picture natural while removing the claimed evidence. Rankings from this metric matched human judgments of good explanations far better than older tests across many models and methods. The authors offer it as a sturdier yardstick for explainability research.

Catalog record

CREATORDanielle Cohen, Hila Chefer, Lior Wolf
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Meaningful Perturbation Metric for Evaluating Explainability Methods Danielle Cohen, Hila Chefer, Lior Wolf Abstract Deep neural networks (DNNs) have demonstrated remarkable success, yet their wide adoption is often hindered by their opaque decision-making. To address this, attribution methods have been proposed to assign relevance values to each part of the input. However, different methods often produce entirely different relevance maps, necessitating the development of standardized metrics to evaluate them. Typically, such evaluation is performed through perturbation, wherein high- or low-relevance regions of the input image are manipulated to examine the change in prediction. In this work, we introduce a novel approach, which harnesses image generation models to perform targeted perturbation. Specifically, we focus on inpainting only the high-relevance pixels of an input image to modify the model’s predictions while preserving image fidelity. This is in contrast to existing approaches, which often produce out-of-distribution modifications, leading to unreliable results. Through extensive experiments, we demonstrate the effectiveness of our approach in generating meaningful rankings across a wide range of models and attribution methods. Crucially, we establish that the ranking produced by our metric exhibits significantly higher correlation with human preferences compared to existing approaches, underscoring its potential for enhancing interpretability in DNNs. 1 Introduction In recent years, deep neural networks (DNNs) have revolutionized research in countless domains. Yet, unlike some predecessors e.g . , decision trees and linear regressors, DNN predictions remain opaque. DNNs often rely on spurious correlations or “shortcuts”, leading to biases [ 6 , 22 ] or robustness issues [ 28 , 29 , 10 ] that hamper their widespread adoption. To overcome this obstacle, several explainability and interpretability techniques have been proposed to clarify the decision-making process of DNNs [ 46 , 50 , 12 , 9 , 24 ] . In this work, we focus on attribution methods, which constitute one of the most prominent approaches for producing such interpretations. Attribution methods assign a corresponding relevance value to each of the input parts (e.g., image pixels), where a higher relevance score suggests a greater impact on the model’s prediction. Various attribution methods have been proposed to interpret the predictions by DNNs, each claiming to be the most “faithful” to the model, while often producing entirely different results than their counterparts. However, in the absence of a rigorous definition of faithfulness and a set of standard acceptable evaluation metrics, it becomes impossible to decipher which attribution method should be employed for a given use case. Figure 1: Examples of OOD predictions following perturbation with (a) pixel deletion [ 30 ] (b) pixel blurring [ 21 ] and (c) per channel mean [ 30 ] on both ResNet [ 27 ] and ViT [ 17 ] backbones. For each image, we randomly draw pixels and apply the perturbation method. We enclose the original and modified prediction. As can be seen, even when the majority of the content is still visible, the model is sensitive to the OOD effect of the perturbation, causing it to modify its prediction. In this work we adopt the dominant notion of faithfulness, where a relevance map is considered to be “faithful” to the model if it reflects the true reasoning process behind the model’s prediction [ 31 , 35 , 9 ] . In other words, given a relevance map, our metric should estimate the correlation between the pixels with high relevance values and the prediction made by the model. Another important parameter for the evaluation of attribution methods is “plausibility”, i.e . , how convincing the explanation is to humans [ 31 ] . Both faithfulness and plausibility are important when evaluating an explanation. Without faithfulness, there is no direct connection between the relevance map and the model we attempt to explain, and therefore the explanation has no utility [ 2 ] . Without plausibility, the explanation may not be clear to humans, which defies its goal of helping humans understand the reasoning behind the decision-making [ 19 ] . The predominant approach to evaluating attribution maps in the context of image classifiers is through image perturbation [ 34 , 30 , 15 , 21 ] . The intuition is that pixels are perturbed from most (least) relevant. If high (low) relevance marks important (unimportant) pixels, perturbing alters (keeps) the prediction. However, existing perturbation methods make unnatural modifications to the input image, such as deleting pixels [ 30 , 35 ] , blurring pixels [ 21 ] or per channel mean [ 30 ] . As we demonstrate in Fig. 1 these alterations often drive the input out of distribution, leading the model to produce predictions that do not correlate with the image pixels. Importantly, in Fig. 1 we apply completely random masks such that the content of the image is still mostly preserved, and even so, the perturbation causes the model to change its prediction. This random baseline demonstrates the significant challenge to disentangling between the OOD effect of the perturbation approach on the prediction and the relevance of the modified pixels to the prediction, making existing perturbation approaches unreliable. Our research hypothesis is that a better approach is to use the perturbed pixels in a way that presents the classifier with a valid alternative. The easiest alternative target is likely to be the class with the second-highest classification score. We employ the capabilities of Stable Diffusion (SD) inpainting [ 42 ] with the appropriate conditioning to modify the high relevance pixels, such that they will align with the second highest prediction, and test whether the prediction changes to the specified target class . This allows us to filter out modifications that lead to unrelated classes, such as the ones in Fig. 1 . Additionally, we add a novel weighting function that compares the relevance maps before and after the inpainting procedure. This validates that the change in prediction was indeed caused by the inpainting procedure, further establishing the connection between the perturbed pixels and the modification in the prediction. We perform extensive experiments to compare our Stratified Inpainting approach to existing perturbation approaches, such as deleting pixels, blurring pixels, and per channel mean. We test different prominent attribution approaches [ 46 , 50 , 1 , 12 , 9 ] , and various image backbones, such as different variants of CNNs [ 27 , 48 ] and Vision Transformers [ 17 ] , and find that in many of the cases, existing methods struggle to provide a meaningful ranking and that the perturbation graphs do not give a meaningful distinction between methods. Surprisingly, we show that the existing perturbation methods often struggle to separate between valid attribution methods and randomly generated relevance maps , further indicating the inadequacy of employing these methods for evaluating explanations. Conversely, our Stratified Inpainting approach maintains decent separation between methods and produces robust results across different models and datasets. Via a user study, we find that the ranking produced by our metric is much more correlated with human preferences, demonstrating that in the case of attribution methods, faithfulness and plausibility are not mutually exclusive, and are both identified by our metric simultaneously. 2 Related Work Explainability in computer vision. Many methods were proposed for generating heatmaps to explain deep networks. Gradient based methods use loss

Read the complete text